2024-03-29T11:22:27.808 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:22:27.808 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710/orig.config.yaml 2024-03-29T11:22:27.875 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:22:27.958 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622710' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHV5Cxb2pgYCJ71n+4eseDt81EMdVU/jdelCc8I0IKUkVzbtbHdsAdlkS3a1HSHERpFUUwZ5c9F6VwCnfo9B8uU= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGlI3AOi5gl6ENZptPZ3nn1/HIM53rjn8vZ9KNQE9ERnA1k0FrtNBNenc01NO2wr+dWLmdyahceIHz3EWoFGyI= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:22:27.958 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:22:27.959 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:22:27.959 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:22:27.960 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:22:27.983 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:22:27.984 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:22:27.984 INFO:teuthology.packaging:ref: None 2024-03-29T11:22:27.984 INFO:teuthology.packaging:tag: None 2024-03-29T11:22:27.984 INFO:teuthology.packaging:branch: main 2024-03-29T11:22:27.984 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:22:27.984 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:22:28.370 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:22:28.372 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:22:28.378 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:22:28.378 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:22:28.425 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:22:28.438 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:22:28.449 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:22:28.475 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:37.026359', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHV5Cxb2pgYCJ71n+4eseDt81EMdVU/jdelCc8I0IKUkVzbtbHdsAdlkS3a1HSHERpFUUwZ5c9F6VwCnfo9B8uU='} 2024-03-29T11:22:28.499 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:37.027963', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGlI3AOi5gl6ENZptPZ3nn1/HIM53rjn8vZ9KNQE9ERnA1k0FrtNBNenc01NO2wr+dWLmdyahceIHz3EWoFGyI='} 2024-03-29T11:22:28.499 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:22:28.505 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:22:28.505 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:22:28.505 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:22:28.624 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd4334260d0>, signals=[15]) 2024-03-29T11:22:28.624 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:22:28.631 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:22:28.631 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:22:28.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:22:28.721 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:22:28.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:22:28.810 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:22:28.817 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-03-29T11:22:28.835 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-03-29T11:22:28.835 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-03-29T11:22:28.891 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-03-29T11:22:28.891 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-03-29T11:22:28.891 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-03-29T11:22:28.892 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:22:28.892 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-03-29T11:22:28.892 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:22:28.892 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:22:28.892 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-03-29T11:22:28.892 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:22:28.892 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:22:28.892 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:22:28.892 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:22:28.892 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:22:28.892 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:22:28.893 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-03-29T11:22:28.917 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-03-29T11:22:28.934 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-03-29T11:22:28.934 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:22:28.991 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:22:28.992 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:22:28.992 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:22:28.992 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-29T11:22:29.009 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:22:29.018 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:22:29.058 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:22:29.058 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:22:29.061 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:22:29.076 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:22:29.084 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:22:29.084 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:22:29.117 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:22:29.207 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:22:29.261 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:22:29.275 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:22:29.276 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:22:29.276 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:22:29.276 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:22:29.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:29.292 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-29T11:22:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:29.293 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-29T11:22:29.349 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:22:29.349 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2024-03-29T11:22:29.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-03-29T11:22:29.352 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:22:29.352 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.14.0-361.el9.x86_64 2024-03-29T11:22:29.352 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2024-03-29T11:22:31.454 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:22:31.661 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:22:32.896 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-29T11:22:33.327 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-29T11:22:33.720 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-29T11:22:34.835 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 6.8 MB/s | 8.0 MB 00:01 2024-03-29T11:22:36.163 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T11:22:36.821 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-29T11:22:44.569 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.1 MB 00:04 2024-03-29T11:22:44.716 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 1.2 MB/s | 6.1 MB 00:05 2024-03-29T11:22:46.020 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-29T11:22:46.347 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-29T11:22:48.180 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-29T11:22:48.190 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-29T11:22:53.173 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-29T11:22:53.179 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T11:22:55.422 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:22:55.422 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:22:55.428 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:22:55.428 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:22:55.514 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:22:55.519 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:22:55.524 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:22:55.525 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T11:22:55.525 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:22:55.525 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:22:55.525 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:22:55.525 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T11:22:55.525 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:22:55.525 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:22:55.525 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:22:55.525 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:22:55.525 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:22:55.526 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:22:55.526 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-03-29T11:22:55.526 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:22:55.526 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 76 M 2024-03-29T11:22:55.526 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 123 M 2024-03-29T11:22:55.526 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:22:55.532 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:22:55.532 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T11:22:55.532 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:22:55.532 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:22:55.532 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:22:55.532 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T11:22:55.533 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:22:55.533 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:22:55.533 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:22:55.533 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:22:55.533 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:22:55.533 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:22:55.533 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-03-29T11:22:55.533 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:22:55.534 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 76 M 2024-03-29T11:22:55.534 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 123 M 2024-03-29T11:22:55.534 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:22:55.683 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:22:55.714 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-03-29T11:22:57.165 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 9.2 MB/s | 15 MB 00:01 2024-03-29T11:22:57.183 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-03-29T11:22:57.507 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T11:22:57.958 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-core-5.14.0-434.el9.x86_64.rpm 6.2 MB/s | 15 MB 00:02 2024-03-29T11:22:58.658 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 11 MB/s | 34 MB 00:03 2024-03-29T11:22:58.661 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:58.662 INFO:teuthology.orchestra.run.smithi183.stdout:Total 24 MB/s | 76 MB 00:03 2024-03-29T11:22:58.662 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:22:59.041 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:22:59.041 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:22:59.140 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 9.3 MB/s | 34 MB 00:03 2024-03-29T11:22:59.143 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:59.143 INFO:teuthology.orchestra.run.smithi155.stdout:Total 21 MB/s | 76 MB 00:03 2024-03-29T11:22:59.143 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:22:59.507 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:22:59.507 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:22:59.882 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:22:59.883 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:23:00.372 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:23:00.372 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:23:02.198 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:23:02.664 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:23:02.798 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:02.884 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:03.329 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:03.423 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:05.634 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:05.779 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:06.225 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:06.366 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:10.514 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:10.560 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:15.295 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:15.807 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:44.450 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:44.450 INFO:teuthology.orchestra.run.smithi183.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:23:44.450 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:23:44.492 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:44.884 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:44.884 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:44.884 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:44.884 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:44.992 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:44.992 INFO:teuthology.orchestra.run.smithi155.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:23:44.992 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:23:45.036 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:23:45.288 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:23:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:23:45.289 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:23:45.448 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:45.448 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:45.448 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:45.448 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:45.646 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:23:45.751 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.751 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:23:45.751 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:23:45.751 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:23:45.752 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:23:45.752 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:23:45.752 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:23:45.752 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:23:45.831 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.832 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:23:45.832 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-03-29T11:23:45.866 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-03-29T11:23:45.866 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-03-29T11:23:45.866 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-03-29T11:23:45.866 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-03-29T11:23:45.866 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-03-29T11:23:45.866 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:23:45.866 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-03-29T11:23:45.866 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-03-29T11:23:45.866 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:23:45.867 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:23:45.868 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-03-29T11:23:45.868 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:23:45.868 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:23:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:23:45.868 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:23:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:23:45.869 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:23:45.870 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:23:45.870 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:23:45.870 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:23:46.202 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:23:46.312 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:46.312 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:23:46.312 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:23:46.312 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:23:46.312 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:23:46.312 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:23:46.312 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:23:46.312 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:23:46.392 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:46.393 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:23:46.393 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2024-03-29T11:23:46.426 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2024-03-29T11:23:46.426 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2024-03-29T11:23:46.426 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.06 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 68.el9 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Group : Unspecified 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 8264254 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : builder@centos.org 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:23:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2024-03-29T11:23:46.428 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2024-03-29T11:23:46.428 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:23:46.428 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:23:46.428 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:23:46.428 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:23:46.428 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2024-03-29T11:23:46.428 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:23:46.428 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:23:46.429 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:23:46.429 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:23:46.429 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:23:46.491 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-03-29T11:23:46.988 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-03-29T11:23:47.884 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:23:47.917 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-03-29T11:23:47.920 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-29T11:23:47.934 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.A89RfvpRmC 2024-03-29T11:23:47.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.A89RfvpRmC 2024-03-29T11:23:48.010 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.A89RfvpRmC 2024-03-29T11:23:48.160 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.A89RfvpRmC is 6KB 2024-03-29T11:23:48.209 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.A89RfvpRmC 2024-03-29T11:23:48.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:23:48.287 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:23:48.288 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:23:48.288 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:23:48.288 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:23:48.289 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:23:48.393 INFO:teuthology.orchestra.run.smithi155.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:23:48.429 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-03-29T11:23:48.431 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-29T11:23:48.443 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-03-29T11:23:48.446 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.xlEztf6RdB 2024-03-29T11:23:48.446 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xlEztf6RdB 2024-03-29T11:23:48.516 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.xlEztf6RdB 2024-03-29T11:23:48.657 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.xlEztf6RdB is 6KB 2024-03-29T11:23:48.706 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.xlEztf6RdB 2024-03-29T11:23:48.721 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:23:48.785 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:23:48.785 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:23:48.785 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:23:48.785 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:23:48.787 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:23:48.956 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-03-29T11:24:18.467 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:24:18.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T11:24:18.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:18.959 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:24:18.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T11:24:18.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:37.279 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-29T11:24:46.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T11:24:46.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:49.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-29T11:25:01.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T11:25:01.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:18.487 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:25:27.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T11:25:27.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:30.927 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-29T11:25:31.360 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T11:25:31.360 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:25:31.360 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-29T11:25:31.374 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:25:31.374 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:25:31.374 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:25:31.374 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:25:32.375 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:25:32.375 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:25:32.441 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-03-29T11:25:32.478 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:01.358 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:26:16.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T11:26:16.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:26:16.757 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-29T11:26:17.143 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T11:26:17.143 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:26:17.143 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-29T11:26:17.159 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:26:17.159 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:26:17.159 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:26:17.159 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:26:18.160 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:26:18.160 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:26:18.244 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-03-29T11:26:18.282 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:18.282 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:26:18.289 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:26:18.290 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:18.292 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:18.311 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:26:18.318 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:26:18.354 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:26:18.355 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:18.357 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:18.440 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:26:18.447 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:26:18.447 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:18.449 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:18.493 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:18.507 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:18.508 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:18.522 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:18.524 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:26:18.532 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:26:18.532 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:18.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:18.564 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:26:18.575 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:26:18.575 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:18.606 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:18.622 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:18.702 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:18.752 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:18.835 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:18.878 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:26:18.878 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:18.944 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:18.985 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:19.039 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:19.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:19.167 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:26:19.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:19.236 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-03-29T11:26:19.238 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-03-29T11:26:19.331 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:19.352 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:19.793 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:26:19.803 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:26:19.803 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:26:19.841 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:26:19.876 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-03-29T11:26:19.955 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-03-29T11:26:19.957 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-03-29T11:26:20.007 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-03-29T11:26:20.009 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:26:20.009 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-03-29T11:26:20.033 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-03-29T11:26:20.034 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-03-29T11:26:20.074 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-03-29T11:26:20.074 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi155.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:26:20.074 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:20.104 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:20.142 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:26:20.142 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:20.185 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:20.226 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:26:20.236 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:26:20.253 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:26:20.254 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi155.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:30:42.403 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi155.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-03-29T11:30:42.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T11:30:42.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:30:42.498 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-29T11:30:42.573 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T11:30:42.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T11:30:42.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:30:42.666 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-29T11:30:42.754 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T11:30:42.755 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:30:42.765 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:30:42.766 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:30:42.767 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:30:42.769 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:30:42.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:30:42.798 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:30:42.815 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:30:42.828 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:30:42.845 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-03-29T11:30:42.845 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:30:42.858 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-03-29T11:30:42.874 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:30:42.875 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-03-29T11:30:42.889 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-03-29T11:30:42.890 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:30:42.906 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:30:42.922 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:30:42.945 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:30:42.950 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:30:42.950 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-29T11:30:42.950 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:42.950 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:42.950 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:42.950 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:42.978 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:30:42.981 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:30:42.981 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-29T11:30:42.982 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:42.982 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:42.982 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:42.982 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:42.982 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:30:42.991 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:30:42.991 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:30:42.991 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:30:42.991 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:30:43.001 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:30:43.001 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:30:43.002 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:30:43.002 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:30:43.005 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:30:43.006 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:30:43.147 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:30:43.148 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:30:43.156 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:30:43.156 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:30:43.585 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:30:43.585 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:30:43.585 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:30:43.588 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:30:43.588 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:30:43.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:30:43.619 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:30:43.619 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:30:43.620 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:30:43.620 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:30:43.690 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:30:43.691 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:30:43.770 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:30:43.772 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:30:43.837 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-03-29T11:30:43.838 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-03-29T11:30:43.839 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-03-29T11:30:43.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-03-29T11:30:44.094 INFO:teuthology.orchestra.run.smithi183.stdout:74 files removed 2024-03-29T11:30:44.102 INFO:teuthology.orchestra.run.smithi155.stdout:74 files removed 2024-03-29T11:30:44.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw 2024-03-29T11:30:44.128 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw 2024-03-29T11:30:44.937 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 161 kB/s | 82 kB 00:00 2024-03-29T11:30:44.955 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 155 kB/s | 82 kB 00:00 2024-03-29T11:30:45.638 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 15 kB/s | 10 kB 00:00 2024-03-29T11:30:45.697 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-29T11:30:46.239 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-29T11:30:46.439 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-29T11:30:46.958 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.7 MB 00:00 2024-03-29T11:30:47.100 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T11:30:48.366 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-29T11:30:48.483 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-29T11:30:49.208 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-29T11:30:50.557 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-03-29T11:30:52.485 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-29T11:30:54.666 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-29T11:30:57.650 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.1 MB 00:01 2024-03-29T11:30:59.917 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.1 MB 00:01 2024-03-29T11:31:00.060 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-29T11:31:01.484 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-29T11:31:01.489 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-29T11:31:03.078 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T11:31:06.344 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-29T11:31:08.036 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T11:31:08.983 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:31:08.998 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:31:08.998 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T11:31:08.998 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:31:08.998 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:31:08.998 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:31:08.998 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-03-29T11:31:08.998 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:31:08.998 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:08.998 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:31:08.999 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:31:09.000 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:31:09.001 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-03-29T11:31:09.001 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:31:09.001 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:31:09.001 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:31:09.001 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:31:09.001 INFO:teuthology.orchestra.run.smithi155.stdout:Install 30 Packages 2024-03-29T11:31:09.001 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2024-03-29T11:31:09.001 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:31:09.001 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 58 M 2024-03-29T11:31:09.001 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:31:09.854 INFO:teuthology.orchestra.run.smithi155.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 6.6 MB/s | 5.3 MB 00:00 2024-03-29T11:31:09.895 INFO:teuthology.orchestra.run.smithi155.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 606 kB/s | 25 kB 00:00 2024-03-29T11:31:10.004 INFO:teuthology.orchestra.run.smithi155.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.7 MB/s | 742 kB 00:00 2024-03-29T11:31:10.163 INFO:teuthology.orchestra.run.smithi155.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.1 MB/s | 498 kB 00:00 2024-03-29T11:31:10.661 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:31:10.664 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:31:10.664 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:31:10.665 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:31:10.666 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:31:10.667 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:31:10.668 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:31:10.668 INFO:teuthology.orchestra.run.smithi183.stdout:Install 30 Packages 2024-03-29T11:31:10.668 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 2 Packages 2024-03-29T11:31:10.668 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:31:10.668 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 58 M 2024-03-29T11:31:10.668 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:31:11.180 INFO:teuthology.orchestra.run.smithi155.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 5.0 MB/s | 5.1 MB 00:01 2024-03-29T11:31:11.238 INFO:teuthology.orchestra.run.smithi155.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 776 kB/s | 45 kB 00:00 2024-03-29T11:31:11.313 INFO:teuthology.orchestra.run.smithi155.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.7 MB/s | 131 kB 00:00 2024-03-29T11:31:11.389 INFO:teuthology.orchestra.run.smithi155.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.1 MB/s | 162 kB 00:00 2024-03-29T11:31:11.472 INFO:teuthology.orchestra.run.smithi155.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.8 MB/s | 320 kB 00:00 2024-03-29T11:31:11.548 INFO:teuthology.orchestra.run.smithi155.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 298 kB 00:00 2024-03-29T11:31:11.614 INFO:teuthology.orchestra.run.smithi155.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 99 kB 00:00 2024-03-29T11:31:11.715 INFO:teuthology.orchestra.run.smithi155.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-29T11:31:11.840 INFO:teuthology.orchestra.run.smithi155.stdout:(13/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 3.5 MB/s | 9.7 MB 00:02 2024-03-29T11:31:11.857 INFO:teuthology.orchestra.run.smithi155.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-29T11:31:11.874 INFO:teuthology.orchestra.run.smithi155.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:31:11.924 INFO:teuthology.orchestra.run.smithi155.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T11:31:11.941 INFO:teuthology.orchestra.run.smithi155.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-03-29T11:31:11.967 INFO:teuthology.orchestra.run.smithi155.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 987 kB/s | 41 kB 00:00 2024-03-29T11:31:11.992 INFO:teuthology.orchestra.run.smithi155.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.2 MB/s | 220 kB 00:00 2024-03-29T11:31:12.009 INFO:teuthology.orchestra.run.smithi155.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 839 kB/s | 35 kB 00:00 2024-03-29T11:31:12.026 INFO:teuthology.orchestra.run.smithi155.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T11:31:12.468 INFO:teuthology.orchestra.run.smithi155.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 674 kB/s | 309 kB 00:00 2024-03-29T11:31:12.560 INFO:teuthology.orchestra.run.smithi155.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-29T11:31:12.660 INFO:teuthology.orchestra.run.smithi155.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-29T11:31:12.760 INFO:teuthology.orchestra.run.smithi155.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 675 kB/s | 67 kB 00:00 2024-03-29T11:31:12.911 INFO:teuthology.orchestra.run.smithi155.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.0 MB/s | 4.4 MB 00:00 2024-03-29T11:31:12.953 INFO:teuthology.orchestra.run.smithi155.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-03-29T11:31:13.011 INFO:teuthology.orchestra.run.smithi155.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-29T11:31:13.128 INFO:teuthology.orchestra.run.smithi155.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-29T11:31:13.895 INFO:teuthology.orchestra.run.smithi183.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.7 MB/s | 5.3 MB 00:03 2024-03-29T11:31:13.995 INFO:teuthology.orchestra.run.smithi183.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-29T11:31:14.287 INFO:teuthology.orchestra.run.smithi183.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 2.5 MB/s | 742 kB 00:00 2024-03-29T11:31:14.312 INFO:teuthology.orchestra.run.smithi155.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 3.8 MB/s | 20 MB 00:05 2024-03-29T11:31:14.487 INFO:teuthology.orchestra.run.smithi183.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.4 MB/s | 498 kB 00:00 2024-03-29T11:31:14.895 INFO:teuthology.orchestra.run.smithi155.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 3.1 MB 00:01 2024-03-29T11:31:15.887 INFO:teuthology.orchestra.run.smithi183.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 9.7 MB 00:05 2024-03-29T11:31:16.004 INFO:teuthology.orchestra.run.smithi183.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 387 kB/s | 45 kB 00:00 2024-03-29T11:31:16.154 INFO:teuthology.orchestra.run.smithi183.stdout:(7/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 3.1 MB/s | 5.1 MB 00:01 2024-03-29T11:31:16.179 INFO:teuthology.orchestra.run.smithi183.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 749 kB/s | 131 kB 00:00 2024-03-29T11:31:16.288 INFO:teuthology.orchestra.run.smithi183.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.2 MB/s | 162 kB 00:00 2024-03-29T11:31:16.547 INFO:teuthology.orchestra.run.smithi183.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 298 kB 00:00 2024-03-29T11:31:16.572 INFO:teuthology.orchestra.run.smithi183.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 817 kB/s | 320 kB 00:00 2024-03-29T11:31:16.680 INFO:teuthology.orchestra.run.smithi183.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 744 kB/s | 99 kB 00:00 2024-03-29T11:31:16.830 INFO:teuthology.orchestra.run.smithi183.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 409 kB/s | 106 kB 00:00 2024-03-29T11:31:16.889 INFO:teuthology.orchestra.run.smithi183.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 767 kB/s | 159 kB 00:00 2024-03-29T11:31:16.906 INFO:teuthology.orchestra.run.smithi183.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-29T11:31:17.048 INFO:teuthology.orchestra.run.smithi183.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 661 kB 00:00 2024-03-29T11:31:17.073 INFO:teuthology.orchestra.run.smithi183.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-03-29T11:31:17.090 INFO:teuthology.orchestra.run.smithi183.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 991 kB/s | 41 kB 00:00 2024-03-29T11:31:17.132 INFO:teuthology.orchestra.run.smithi183.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2024-03-29T11:31:17.149 INFO:teuthology.orchestra.run.smithi183.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 595 kB/s | 35 kB 00:00 2024-03-29T11:31:17.183 INFO:teuthology.orchestra.run.smithi183.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 918 kB/s | 46 kB 00:00 2024-03-29T11:31:17.558 INFO:teuthology.orchestra.run.smithi183.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 755 kB/s | 309 kB 00:00 2024-03-29T11:31:17.650 INFO:teuthology.orchestra.run.smithi183.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-03-29T11:31:17.700 INFO:teuthology.orchestra.run.smithi183.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2024-03-29T11:31:17.768 INFO:teuthology.orchestra.run.smithi183.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-29T11:31:17.926 INFO:teuthology.orchestra.run.smithi183.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 838 kB 00:00 2024-03-29T11:31:17.976 INFO:teuthology.orchestra.run.smithi183.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-03-29T11:31:18.218 INFO:teuthology.orchestra.run.smithi183.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 6.6 MB/s | 1.6 MB 00:00 2024-03-29T11:31:18.335 INFO:teuthology.orchestra.run.smithi183.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 4.4 MB 00:01 2024-03-29T11:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 2.4 MB/s | 20 MB 00:08 2024-03-29T11:31:19.769 INFO:teuthology.orchestra.run.smithi183.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.2 MB/s | 3.4 MB 00:01 2024-03-29T11:31:20.970 INFO:teuthology.orchestra.run.smithi155.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 434 kB/s | 3.4 MB 00:07 2024-03-29T11:31:20.973 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:20.973 INFO:teuthology.orchestra.run.smithi155.stdout:Total 4.8 MB/s | 58 MB 00:11 2024-03-29T11:31:20.974 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:31:21.013 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:31:21.013 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:31:21.027 INFO:teuthology.orchestra.run.smithi183.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 3.1 MB 00:02 2024-03-29T11:31:21.032 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:21.033 INFO:teuthology.orchestra.run.smithi183.stdout:Total 5.6 MB/s | 58 MB 00:10 2024-03-29T11:31:21.033 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:31:21.068 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:31:21.068 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:31:21.480 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:31:21.480 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:31:21.531 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:31:21.531 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:31:22.935 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:31:22.994 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:31:23.027 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:31:23.058 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:31:23.078 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:31:23.083 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:31:23.106 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:23.115 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:31:23.122 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:23.140 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:31:23.163 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:23.180 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:23.315 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:23.347 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:23.371 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:23.402 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:23.714 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:31:23.728 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:31:23.760 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:23.793 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:23.843 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:23.908 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:23.924 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:31:23.953 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:23.998 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:31:23.998 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:24.026 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:24.027 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:31:24.071 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:31:24.089 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:24.117 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:31:24.140 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:24.161 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:31:24.163 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:24.227 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:24.229 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:24.254 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:24.261 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:31:24.317 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:24.352 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:31:24.389 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:31:24.430 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:31:24.458 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:31:24.470 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:31:24.486 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:31:24.502 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:31:24.527 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:31:24.531 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:31:24.559 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:31:24.564 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:31:24.588 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:31:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:31:24.883 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:31:24.911 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:24.933 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:31:24.967 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:24.984 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:25.016 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:31:25.056 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:31:25.057 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:25.088 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:31:25.128 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:31:25.506 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:31:25.530 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:25.578 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:31:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:25.603 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:25.669 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:31:25.674 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:31:25.742 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:31:25.772 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:31:27.777 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:27.829 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:28.377 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:28.415 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:28.496 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:28.496 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:31:28.496 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:31:28.496 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:31:28.525 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:28.533 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:28.571 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:28.660 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:28.660 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:31:28.660 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:31:28.660 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:31:28.723 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:38.050 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:38.050 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-03-29T11:31:38.050 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-03-29T11:31:38.051 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-03-29T11:31:38.051 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-03-29T11:31:38.051 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-03-29T11:31:38.051 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-03-29T11:31:38.051 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-03-29T11:31:38.051 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:31:38.273 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:38.273 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-03-29T11:31:38.273 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-03-29T11:31:38.273 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-03-29T11:31:38.274 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-03-29T11:31:38.274 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-03-29T11:31:38.274 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-03-29T11:31:38.274 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-03-29T11:31:38.274 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:31:38.951 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:38.975 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:38.975 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:38.975 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:31:38.976 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:38.976 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:38.976 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:31:38.977 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:39.046 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:39.046 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:39.159 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:39.183 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:39.183 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:39.183 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:31:39.183 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:39.183 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:39.183 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:31:39.185 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:39.269 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:39.270 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:31:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:31:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:31:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:31:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:31:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:31:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:31:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:31:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:31:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:31:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:31:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:31:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:31:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:31:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:31:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:31:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:31:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:31:40.421 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:40.421 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:31:40.421 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:31:40.421 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:31:40.421 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:31:40.421 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:31:40.421 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:31:40.421 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:40.421 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:31:40.421 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:31:40.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:31:40.423 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:31:40.423 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:31:40.423 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:31:40.423 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:31:40.424 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:31:40.424 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:31:40.424 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:31:40.424 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:40.424 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:31:40.424 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:31:41.207 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:41.207 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:31:41.207 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-03-29T11:31:41.207 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:31:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:31:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:31:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:31:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:31:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:31:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:31:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:31:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:31:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:31:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:31:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:31:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:31:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:31:41.210 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:31:41.322 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:41.322 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:31:41.322 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-03-29T11:31:41.323 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.323 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.323 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:31:41.323 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:31:41.323 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.323 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.323 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.323 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.323 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:31:41.323 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:31:41.323 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:31:41.324 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.324 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:31:41.324 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:31:41.324 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:31:41.324 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.324 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:31:41.324 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.324 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:31:41.324 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:31:41.324 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:31:41.325 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:31:41.326 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:31:41.326 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:31:41.541 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test 2024-03-29T11:31:41.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test 2024-03-29T11:31:42.104 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:36 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:31:42.245 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:35 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:31:42.262 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:31:42.263 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:31:42.263 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T11:31:42.264 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:31:42.264 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:31:42.264 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:31:42.264 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T11:31:42.264 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:31:42.264 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:31:42.264 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:31:42.265 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:31:42.265 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:31:42.265 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:31:42.265 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:31:42.265 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:31:42.265 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-03-29T11:31:42.265 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:31:42.266 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 48 M 2024-03-29T11:31:42.267 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 204 M 2024-03-29T11:31:42.267 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:31:42.394 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:31:42.396 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:31:42.396 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T11:31:42.396 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:31:42.396 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:31:42.396 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:31:42.397 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T11:31:42.397 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:31:42.397 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:31:42.397 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:31:42.397 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:31:42.397 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:31:42.397 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:31:42.397 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:31:42.397 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:31:42.397 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-03-29T11:31:42.397 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:31:42.398 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 48 M 2024-03-29T11:31:42.398 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 204 M 2024-03-29T11:31:42.398 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:31:42.543 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 986 kB/s | 217 kB 00:00 2024-03-29T11:31:42.601 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-29T11:31:42.638 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:31:42.731 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T11:31:42.783 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T11:31:42.815 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:31:42.983 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-29T11:31:43.084 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 214 kB/s | 163 kB 00:00 2024-03-29T11:31:43.266 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 228 kB/s | 163 kB 00:00 2024-03-29T11:31:43.368 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 255 kB/s | 187 kB 00:00 2024-03-29T11:32:03.931 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 47 MB 00:21 2024-03-29T11:32:03.934 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:03.934 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.2 MB/s | 48 MB 00:21 2024-03-29T11:32:03.948 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:32:03.969 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:32:03.969 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:32:04.346 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:32:04.348 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:32:04.931 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:32:04.997 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:05.048 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:05.088 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:05.123 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:05.150 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:09.615 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:12.028 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:12.028 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:12.028 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:12.029 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:12.029 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:12.029 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:12.398 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:12.398 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:12.398 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:32:12.398 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.398 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:12.399 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:12.399 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:12.399 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:12.399 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:12.399 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:12.399 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:32:12.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph 2024-03-29T11:32:13.284 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:32:13.445 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:32:13.447 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-29T11:32:13.447 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T11:32:13.448 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-29T11:32:13.448 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:32:13.448 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:13.448 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T11:32:13.448 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:13.449 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:13.449 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:13.449 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:13.449 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:13.449 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:13.449 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:13.449 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:13.450 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:13.450 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:13.450 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:13.450 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:13.450 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:13.450 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:13.450 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:13.451 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:13.451 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:13.451 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:13.451 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:13.451 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:13.451 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:13.451 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:13.451 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:13.451 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:13.452 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:32:13.453 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-29T11:32:13.453 INFO:teuthology.orchestra.run.smithi183.stdout:Install 36 Packages 2024-03-29T11:32:13.453 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:13.454 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 30 M 2024-03-29T11:32:13.454 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 107 M 2024-03-29T11:32:13.454 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:32:14.027 INFO:teuthology.orchestra.run.smithi183.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 21 kB/s | 6.6 kB 00:00 2024-03-29T11:32:14.324 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 47 MB 00:32 2024-03-29T11:32:14.327 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:14.327 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.5 MB/s | 48 MB 00:32 2024-03-29T11:32:14.346 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:32:14.364 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:32:14.364 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:32:14.751 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:32:14.751 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:32:14.994 INFO:teuthology.orchestra.run.smithi183.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-03-29T11:32:15.338 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:32:15.425 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:15.463 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:15.495 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:15.539 INFO:teuthology.orchestra.run.smithi183.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 2.1 MB 00:01 2024-03-29T11:32:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:15.745 INFO:teuthology.orchestra.run.smithi183.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.2 MB/s | 247 kB 00:00 2024-03-29T11:32:15.862 INFO:teuthology.orchestra.run.smithi183.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:32:15.904 INFO:teuthology.orchestra.run.smithi183.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:32:15.929 INFO:teuthology.orchestra.run.smithi183.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T11:32:15.959 INFO:teuthology.orchestra.run.smithi183.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-29T11:32:15.979 INFO:teuthology.orchestra.run.smithi183.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-29T11:32:16.021 INFO:teuthology.orchestra.run.smithi183.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:32:16.097 INFO:teuthology.orchestra.run.smithi183.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:32:16.131 INFO:teuthology.orchestra.run.smithi183.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:32:16.164 INFO:teuthology.orchestra.run.smithi183.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:32:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-29T11:32:16.290 INFO:teuthology.orchestra.run.smithi183.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-03-29T11:32:16.865 INFO:teuthology.orchestra.run.smithi183.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-03-29T11:32:17.207 INFO:teuthology.orchestra.run.smithi183.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 443 kB/s | 151 kB 00:00 2024-03-29T11:32:17.299 INFO:teuthology.orchestra.run.smithi183.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 474 kB/s | 43 kB 00:00 2024-03-29T11:32:17.399 INFO:teuthology.orchestra.run.smithi183.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-29T11:32:17.525 INFO:teuthology.orchestra.run.smithi183.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-29T11:32:17.592 INFO:teuthology.orchestra.run.smithi183.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-29T11:32:17.659 INFO:teuthology.orchestra.run.smithi183.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 266 kB/s | 18 kB 00:00 2024-03-29T11:32:17.734 INFO:teuthology.orchestra.run.smithi183.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 309 kB/s | 23 kB 00:00 2024-03-29T11:32:17.801 INFO:teuthology.orchestra.run.smithi183.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 292 kB/s | 19 kB 00:00 2024-03-29T11:32:17.885 INFO:teuthology.orchestra.run.smithi183.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 236 kB/s | 20 kB 00:00 2024-03-29T11:32:17.951 INFO:teuthology.orchestra.run.smithi183.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 696 kB/s | 46 kB 00:00 2024-03-29T11:32:18.018 INFO:teuthology.orchestra.run.smithi183.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-29T11:32:18.127 INFO:teuthology.orchestra.run.smithi183.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-29T11:32:18.194 INFO:teuthology.orchestra.run.smithi183.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 247 kB/s | 16 kB 00:00 2024-03-29T11:32:18.294 INFO:teuthology.orchestra.run.smithi183.stdout:(30/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 4.7 MB 00:04 2024-03-29T11:32:18.319 INFO:teuthology.orchestra.run.smithi183.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 717 kB/s | 90 kB 00:00 2024-03-29T11:32:18.353 INFO:teuthology.orchestra.run.smithi183.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 613 kB/s | 36 kB 00:00 2024-03-29T11:32:18.387 INFO:teuthology.orchestra.run.smithi183.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-29T11:32:18.445 INFO:teuthology.orchestra.run.smithi183.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-29T11:32:18.462 INFO:teuthology.orchestra.run.smithi183.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-03-29T11:32:20.042 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:20.126 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:22.384 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:22.385 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:22.385 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:22.385 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:22.385 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:22.385 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:22.761 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:32:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:22.762 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:32:23.065 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph 2024-03-29T11:32:23.640 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:32:23.802 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:32:23.804 INFO:teuthology.orchestra.run.smithi155.stdout:========================================================================================= 2024-03-29T11:32:23.805 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T11:32:23.805 INFO:teuthology.orchestra.run.smithi155.stdout:========================================================================================= 2024-03-29T11:32:23.805 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:32:23.805 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:23.805 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T11:32:23.805 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:23.806 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:23.806 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:23.806 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:23.806 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:23.806 INFO:teuthology.orchestra.run.smithi155.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:23.806 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:23.806 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:23.806 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:23.806 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:23.807 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:23.808 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:23.809 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:23.809 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:23.809 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:32:23.809 INFO:teuthology.orchestra.run.smithi155.stdout:========================================================================================= 2024-03-29T11:32:23.809 INFO:teuthology.orchestra.run.smithi155.stdout:Install 36 Packages 2024-03-29T11:32:23.809 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:23.810 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 30 M 2024-03-29T11:32:23.810 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 107 M 2024-03-29T11:32:23.810 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:32:24.554 INFO:teuthology.orchestra.run.smithi155.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-29T11:32:25.896 INFO:teuthology.orchestra.run.smithi155.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 870 kB/s | 1.5 MB 00:01 2024-03-29T11:32:27.938 INFO:teuthology.orchestra.run.smithi155.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 4.7 MB 00:03 2024-03-29T11:32:28.146 INFO:teuthology.orchestra.run.smithi155.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.2 MB/s | 247 kB 00:00 2024-03-29T11:32:28.271 INFO:teuthology.orchestra.run.smithi155.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T11:32:28.313 INFO:teuthology.orchestra.run.smithi155.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:32:28.339 INFO:teuthology.orchestra.run.smithi155.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T11:32:28.364 INFO:teuthology.orchestra.run.smithi155.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-29T11:32:28.407 INFO:teuthology.orchestra.run.smithi155.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 929 kB/s | 39 kB 00:00 2024-03-29T11:32:28.448 INFO:teuthology.orchestra.run.smithi155.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:32:28.526 INFO:teuthology.orchestra.run.smithi155.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:32:28.565 INFO:teuthology.orchestra.run.smithi155.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.6 MB/s | 103 kB 00:00 2024-03-29T11:32:28.599 INFO:teuthology.orchestra.run.smithi155.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-29T11:32:28.641 INFO:teuthology.orchestra.run.smithi155.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T11:32:28.766 INFO:teuthology.orchestra.run.smithi155.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 999 kB/s | 125 kB 00:00 2024-03-29T11:32:28.841 INFO:teuthology.orchestra.run.smithi155.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 298 kB/s | 22 kB 00:00 2024-03-29T11:32:29.067 INFO:teuthology.orchestra.run.smithi155.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 671 kB/s | 151 kB 00:00 2024-03-29T11:32:29.151 INFO:teuthology.orchestra.run.smithi155.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 521 kB/s | 43 kB 00:00 2024-03-29T11:32:29.284 INFO:teuthology.orchestra.run.smithi155.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-29T11:32:29.409 INFO:teuthology.orchestra.run.smithi155.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-29T11:32:29.468 INFO:teuthology.orchestra.run.smithi155.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-29T11:32:29.560 INFO:teuthology.orchestra.run.smithi155.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-29T11:32:29.619 INFO:teuthology.orchestra.run.smithi155.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 398 kB/s | 23 kB 00:00 2024-03-29T11:32:29.694 INFO:teuthology.orchestra.run.smithi155.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-03-29T11:32:29.769 INFO:teuthology.orchestra.run.smithi155.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 263 kB/s | 20 kB 00:00 2024-03-29T11:32:29.828 INFO:teuthology.orchestra.run.smithi155.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 795 kB/s | 46 kB 00:00 2024-03-29T11:32:29.903 INFO:teuthology.orchestra.run.smithi155.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-29T11:32:29.987 INFO:teuthology.orchestra.run.smithi155.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-29T11:32:30.061 INFO:teuthology.orchestra.run.smithi155.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 223 kB/s | 16 kB 00:00 2024-03-29T11:32:30.146 INFO:teuthology.orchestra.run.smithi155.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-03-29T11:32:30.219 INFO:teuthology.orchestra.run.smithi183.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 17 MB 00:15 2024-03-29T11:32:30.221 INFO:teuthology.orchestra.run.smithi155.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 478 kB/s | 36 kB 00:00 2024-03-29T11:32:30.224 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:30.225 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.8 MB/s | 30 MB 00:16 2024-03-29T11:32:30.249 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:32:30.292 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:32:30.292 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:32:30.305 INFO:teuthology.orchestra.run.smithi155.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-29T11:32:30.364 INFO:teuthology.orchestra.run.smithi155.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-03-29T11:32:30.456 INFO:teuthology.orchestra.run.smithi155.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-29T11:32:30.809 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:32:30.809 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:32:32.103 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:32:32.172 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:32:32.221 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:32:32.258 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:32:32.332 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:32:32.366 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:32:32.399 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:32:32.493 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:32:32.581 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:32:32.622 INFO:teuthology.orchestra.run.smithi155.stdout:(35/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86 256 kB/s | 2.1 MB 00:08 2024-03-29T11:32:32.627 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:32:32.703 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:32:32.753 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:32:32.799 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:32:32.856 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:32:32.912 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:32:33.064 INFO:teuthology.orchestra.run.smithi155.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 2.3 MB/s | 17 MB 00:07 2024-03-29T11:32:33.069 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:33.069 INFO:teuthology.orchestra.run.smithi155.stdout:Total 3.2 MB/s | 30 MB 00:09 2024-03-29T11:32:33.093 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:32:33.105 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:32:33.136 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:32:33.136 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:32:33.153 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:32:33.192 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:32:33.226 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:32:33.257 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:32:33.335 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:32:33.366 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:33.400 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:32:33.441 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:32:33.475 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:32:33.501 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:32:33.553 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:32:33.553 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:32:33.586 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:32:33.624 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:32:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:32:33.839 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:32:33.963 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:34.124 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:32:34.145 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:34.170 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:34.171 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:34.171 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:32:34.171 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:34.171 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:34.171 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:34.892 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:32:34.950 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:32:34.987 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:32:35.029 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:32:35.098 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:32:35.136 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:32:35.168 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:32:35.259 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:32:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:32:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:32:35.450 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:32:35.504 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:32:35.543 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:32:35.592 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:32:35.612 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:35.638 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:35.639 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:35.639 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:32:35.639 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:35.639 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:35.639 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:35.655 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:32:35.839 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:32:35.880 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:32:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:32:35.946 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:32:35.975 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:32:36.052 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:32:36.086 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:36.088 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:36.111 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:36.112 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:36.112 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:32:36.112 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:36.112 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:36.112 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:36.119 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:32:36.159 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:32:36.194 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:32:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:32:36.318 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:32:36.322 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:36.344 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:36.345 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:36.345 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:32:36.345 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:36.345 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:36.345 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:36.360 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:32:36.411 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:36.432 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:32:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:32:36.694 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:36.852 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:32:36.882 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:36.907 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:36.907 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:32:36.907 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:36.908 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:38.065 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:38.066 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:32:38.066 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:32:38.066 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:32:38.066 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:32:38.066 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:32:38.067 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:32:38.067 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:32:38.067 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:32:38.067 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:32:38.067 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:32:38.067 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:32:38.068 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:32:38.068 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:32:38.068 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:32:38.068 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:32:38.069 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:32:38.069 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:32:38.069 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:32:38.069 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:32:38.069 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:32:38.069 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:38.070 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:32:38.070 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:32:38.070 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:32:38.070 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:32:38.070 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:32:38.070 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:32:38.071 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:32:38.071 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:32:38.072 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:38.072 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:32:38.072 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:32:38.072 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:32:38.072 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:32:38.072 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:32:38.342 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:38.368 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:38.368 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:38.368 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:32:38.368 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:38.368 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:38.368 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:38.818 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:38.841 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:38.842 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:38.842 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:32:38.842 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:38.842 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:38.842 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:38.972 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:32:38.972 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:32:38.973 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:32:38.974 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:38.975 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:32:39.060 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:39.083 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:39.083 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:39.083 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:32:39.083 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:39.083 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:39.084 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:39.149 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:39.307 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base 2024-03-29T11:32:39.880 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:32:39.978 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:40.031 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:32:40.032 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T11:32:40.032 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:32:40.100 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install cephadm 2024-03-29T11:32:40.662 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:32:40.812 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:32:40.813 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:32:40.813 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T11:32:40.813 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:32:40.813 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:32:40.813 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:32:40.813 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:40.813 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:32:40.813 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:32:40.814 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T11:32:40.814 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:40.814 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 747 k 2024-03-29T11:32:40.814 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 754 k 2024-03-29T11:32:40.814 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:32:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:32:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:32:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:32:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:32:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:32:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:32:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:32:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:32:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:32:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:32:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:32:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:32:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:32:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:32:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:32:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:32:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:32:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:32:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:32:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:32:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:32:41.505 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.1 MB/s | 747 kB 00:00 2024-03-29T11:32:41.505 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:41.506 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.1 MB/s | 747 kB 00:00 2024-03-29T11:32:41.506 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:32:41.511 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:32:41.511 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:32:41.516 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:32:41.517 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:32:41.590 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:32:41.791 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:32:41.791 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:41.791 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:32:41.791 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:41.791 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:41.792 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:41.792 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:41.792 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:41.792 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:41.792 INFO:teuthology.orchestra.run.smithi155.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:32:41.792 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:32:41.792 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:32:41.792 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:32:41.792 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:32:41.792 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:32:41.792 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:32:41.793 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:32:41.793 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:32:41.793 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:32:41.793 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:32:41.793 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:32:41.793 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:32:41.793 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:32:41.793 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:32:41.793 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:32:41.793 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:32:41.794 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:32:41.794 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:32:41.794 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:32:41.794 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:32:41.794 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:32:41.794 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:32:41.794 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:32:41.794 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:32:41.794 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:32:41.795 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:32:41.795 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:32:41.795 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:32:41.795 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:32:41.795 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:41.795 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:32:42.064 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:42.111 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base 2024-03-29T11:32:42.161 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:42.562 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:42.700 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:32:42.800 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:42.852 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:32:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:42.852 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:32:42.852 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:32:42.853 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T11:32:42.853 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:32:42.925 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install cephadm 2024-03-29T11:32:43.052 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:32:43.492 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:32:43.625 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:32:43.644 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:32:43.645 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:32:43.645 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T11:32:43.645 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:32:43.645 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:32:43.645 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:32:43.645 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:43.645 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:32:43.645 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:32:43.645 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T11:32:43.645 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:43.646 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 747 k 2024-03-29T11:32:43.646 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 754 k 2024-03-29T11:32:43.646 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:32:43.778 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 145 k 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 436 k 2024-03-29T11:32:43.779 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:32:44.545 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 190 kB/s | 145 kB 00:00 2024-03-29T11:32:44.545 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:44.545 INFO:teuthology.orchestra.run.smithi183.stdout:Total 190 kB/s | 145 kB 00:00 2024-03-29T11:32:44.546 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:32:44.553 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:32:44.553 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:32:44.606 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:32:44.607 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:32:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:32:44.821 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:44.831 INFO:teuthology.orchestra.run.smithi155.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 631 kB/s | 747 kB 00:01 2024-03-29T11:32:44.831 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:44.832 INFO:teuthology.orchestra.run.smithi155.stdout:Total 630 kB/s | 747 kB 00:01 2024-03-29T11:32:44.832 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:32:44.838 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:32:44.838 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:32:44.844 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:32:44.845 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:32:44.845 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:44.845 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:44.845 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:32:44.845 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:44.923 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:32:45.352 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:45.470 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:45.785 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:45.785 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:45.785 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:32:45.785 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:45.785 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:45.786 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:32:45.860 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:46.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr 2024-03-29T11:32:46.127 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:46.128 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:46.128 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:32:46.128 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:46.128 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:46.128 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:32:46.328 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:32:46.634 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:32:46.733 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:46.785 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:32:46.786 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T11:32:46.786 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:32:46.857 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:32:46.902 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:32:47.057 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:32:47.057 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 145 k 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 436 k 2024-03-29T11:32:47.058 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:32:47.280 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 658 kB/s | 145 kB 00:00 2024-03-29T11:32:47.280 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:47.280 INFO:teuthology.orchestra.run.smithi155.stdout:Total 653 kB/s | 145 kB 00:00 2024-03-29T11:32:47.281 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:32:47.288 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:32:47.288 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:32:47.340 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:32:47.341 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:32:47.418 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:32:47.478 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:32:47.564 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:47.569 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:32:47.570 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-29T11:32:47.570 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T11:32:47.570 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-29T11:32:47.570 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:32:47.570 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:47.571 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:32:47.572 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-29T11:32:47.572 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-03-29T11:32:47.572 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:47.572 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.9 M 2024-03-29T11:32:47.572 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-03-29T11:32:47.572 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:32:47.588 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:47.588 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:47.588 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:32:47.588 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:48.449 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 61 kB/s | 15 kB 00:00 2024-03-29T11:32:48.474 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 113 kB/s | 31 kB 00:00 2024-03-29T11:32:48.544 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:48.544 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:48.544 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:32:48.544 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:48.544 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:48.544 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:32:48.608 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-29T11:32:48.650 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.9 MB/s | 1.8 MB 00:00 2024-03-29T11:32:48.733 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-03-29T11:32:48.775 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr 2024-03-29T11:32:48.875 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 138 kB/s | 19 kB 00:00 2024-03-29T11:32:48.900 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-29T11:32:48.984 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-03-29T11:32:49.009 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-29T11:32:49.060 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.9 MB/s | 863 kB 00:00 2024-03-29T11:32:49.085 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 409 kB/s | 31 kB 00:00 2024-03-29T11:32:49.110 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-29T11:32:49.115 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:49.115 INFO:teuthology.orchestra.run.smithi183.stdout:Total 4.4 MB/s | 6.9 MB 00:01 2024-03-29T11:32:49.134 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:32:49.151 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:32:49.151 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:32:49.319 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:32:49.320 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:32:49.356 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:32:49.455 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:49.508 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:32:49.509 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T11:32:49.509 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:32:49.583 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:32:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:32:49.902 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:32:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:32:49.985 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:32:50.015 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:32:50.116 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:32:50.151 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:32:50.240 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:32:50.302 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:32:50.303 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================== 2024-03-29T11:32:50.303 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T11:32:50.303 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================== 2024-03-29T11:32:50.303 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:32:50.303 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:32:50.303 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T11:32:50.303 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:32:50.303 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:32:50.303 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:32:50.303 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:32:50.303 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:32:50.304 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:32:50.304 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:32:50.304 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:32:50.304 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:32:50.304 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:32:50.304 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:32:50.304 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:50.304 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:32:50.304 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================== 2024-03-29T11:32:50.304 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-03-29T11:32:50.304 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:50.305 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 6.9 M 2024-03-29T11:32:50.305 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2024-03-29T11:32:50.305 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:32:50.343 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:32:50.386 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:32:50.414 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:32:50.446 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:32:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:32:50.721 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:32:50.852 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 166 kB/s | 31 kB 00:00 2024-03-29T11:32:50.868 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 74 kB/s | 15 kB 00:00 2024-03-29T11:32:51.019 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T11:32:51.061 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-29T11:32:51.103 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.0 MB/s | 1.8 MB 00:00 2024-03-29T11:32:51.153 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 391 kB/s | 19 kB 00:00 2024-03-29T11:32:51.220 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.4 MB/s | 548 kB 00:00 2024-03-29T11:32:51.262 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.5 MB/s | 863 kB 00:00 2024-03-29T11:32:51.279 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-03-29T11:32:51.321 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-03-29T11:32:51.363 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 306 kB/s | 31 kB 00:00 2024-03-29T11:32:51.414 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.8 MB/s | 2.0 MB 00:00 2024-03-29T11:32:51.416 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:51.416 INFO:teuthology.orchestra.run.smithi155.stdout:Total 6.2 MB/s | 6.9 MB 00:01 2024-03-29T11:32:51.434 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:32:51.452 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:32:51.452 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:32:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:32:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:32:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:32:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:32:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:32:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:32:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:32:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:32:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:32:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:32:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:32:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:32:51.623 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:32:51.623 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:32:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:32:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:32:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:32:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:51.877 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:32:52.131 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:32:52.140 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:32:52.222 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:32:52.258 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:32:52.295 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:32:52.326 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:32:52.425 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:32:52.552 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:32:52.654 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:32:52.696 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:32:52.697 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:32:52.726 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:32:52.758 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:32:52.853 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:32:52.854 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-29T11:32:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T11:32:52.854 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-29T11:32:52.855 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:32:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:32:52.855 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T11:32:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:32:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:32:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:32:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:32:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:32:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:32:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:32:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:32:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:32:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:32:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:32:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:52.856 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:32:52.856 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-29T11:32:52.856 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-03-29T11:32:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:32:52.857 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 37 M 2024-03-29T11:32:52.857 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 209 M 2024-03-29T11:32:52.857 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:32:52.993 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:32:53.024 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:32:53.101 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 221 kB/s | 32 kB 00:00 2024-03-29T11:32:53.134 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2024-03-29T11:32:53.176 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 34 kB 00:00 2024-03-29T11:32:53.243 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-29T11:32:53.418 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T11:32:53.485 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-29T11:32:53.510 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:32:53.686 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-29T11:32:53.836 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-29T11:32:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:32:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:32:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:32:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:32:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:32:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:32:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:32:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:32:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:32:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:32:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:32:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:32:53.861 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T11:32:54.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:32:54.193 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:32:54.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:32:54.195 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:54.195 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:32:54.361 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 5.3 MB/s | 7.4 MB 00:01 2024-03-29T11:32:54.386 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 446 kB/s | 245 kB 00:00 2024-03-29T11:32:54.389 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:54.389 INFO:teuthology.orchestra.run.smithi183.stdout:Total 24 MB/s | 37 MB 00:01 2024-03-29T11:32:54.430 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:32:54.453 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:32:54.453 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:32:54.456 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:32:54.818 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:32:54.818 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:32:55.024 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:32:55.179 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:32:55.180 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-29T11:32:55.180 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T11:32:55.180 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-29T11:32:55.180 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:32:55.180 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:32:55.180 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:32:55.181 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-29T11:32:55.182 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-03-29T11:32:55.182 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:32:55.182 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 37 M 2024-03-29T11:32:55.182 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 209 M 2024-03-29T11:32:55.182 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:32:55.509 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:32:55.592 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:32:55.624 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:32:55.646 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 242 kB/s | 32 kB 00:00 2024-03-29T11:32:55.679 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 521 kB/s | 17 kB 00:00 2024-03-29T11:32:55.715 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:32:55.738 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-29T11:32:55.752 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:32:55.763 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 424 kB/s | 34 kB 00:00 2024-03-29T11:32:55.919 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:32:55.951 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:32:55.955 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-29T11:32:56.022 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T11:32:56.035 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:32:56.047 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:32:56.173 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.4 MB/s | 803 kB 00:00 2024-03-29T11:32:56.348 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-29T11:32:56.390 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 910 kB/s | 195 kB 00:00 2024-03-29T11:32:56.590 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-29T11:32:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:32:56.813 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:32:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:32:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:32:59.101 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:02.068 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:02.068 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:02.070 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:02.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:02.991 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:03.143 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:03.144 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T11:33:03.144 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T11:33:03.144 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T11:33:03.144 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:33:03.144 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:03.144 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T11:33:03.144 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:03.144 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T11:33:03.145 INFO:teuthology.orchestra.run.smithi183.stdout:Install 13 Packages 2024-03-29T11:33:03.146 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:03.146 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.0 M 2024-03-29T11:33:03.146 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-03-29T11:33:03.146 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:33:03.601 INFO:teuthology.orchestra.run.smithi183.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 247 kB/s | 24 kB 00:00 2024-03-29T11:33:03.643 INFO:teuthology.orchestra.run.smithi183.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 128 kB/s | 18 kB 00:00 2024-03-29T11:33:03.710 INFO:teuthology.orchestra.run.smithi183.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-29T11:33:03.735 INFO:teuthology.orchestra.run.smithi183.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-29T11:33:03.760 INFO:teuthology.orchestra.run.smithi183.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-29T11:33:03.777 INFO:teuthology.orchestra.run.smithi183.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T11:33:04.019 INFO:teuthology.orchestra.run.smithi183.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 101 kB/s | 52 kB 00:00 2024-03-29T11:33:04.036 INFO:teuthology.orchestra.run.smithi183.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-29T11:33:04.053 INFO:teuthology.orchestra.run.smithi183.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-03-29T11:33:04.254 INFO:teuthology.orchestra.run.smithi183.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 866 kB/s | 203 kB 00:00 2024-03-29T11:33:04.337 INFO:teuthology.orchestra.run.smithi183.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-29T11:33:04.362 INFO:teuthology.orchestra.run.smithi183.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 191 kB/s | 59 kB 00:00 2024-03-29T11:33:04.396 INFO:teuthology.orchestra.run.smithi183.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-03-29T11:33:04.400 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:04.400 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-29T11:33:04.400 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:33:04.413 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:33:04.413 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:33:04.502 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:33:04.502 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:33:05.012 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:33:05.166 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:05.222 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:05.286 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:05.323 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:05.365 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:05.467 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:05.575 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:05.633 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:06.351 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:06.397 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:06.430 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:06.479 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:06.525 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:07.299 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:07.299 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:07.299 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:07.299 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:07.300 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:07.300 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:07.300 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:07.300 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:07.300 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:07.300 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:07.300 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:07.300 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:07.300 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:07.339 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 640 kB/s | 7.4 MB 00:11 2024-03-29T11:33:07.342 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:07.343 INFO:teuthology.orchestra.run.smithi155.stdout:Total 3.0 MB/s | 37 MB 00:12 2024-03-29T11:33:07.386 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:33:07.413 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:33:07.413 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:33:07.683 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:07.683 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:07.683 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:33:07.683 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:07.683 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:07.683 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:07.683 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:07.683 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:07.683 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:07.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:07.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:07.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:07.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:07.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:07.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:07.684 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:07.684 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:07.684 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:07.769 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:33:07.770 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:33:07.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:33:08.507 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:08.537 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:33:08.577 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:33:08.659 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:08.660 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T11:33:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T11:33:08.660 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T11:33:08.660 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:33:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:08.660 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T11:33:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:08.661 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:33:08.661 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T11:33:08.661 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-03-29T11:33:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:08.661 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.8 M 2024-03-29T11:33:08.662 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 33 M 2024-03-29T11:33:08.662 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:33:08.671 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:08.722 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:33:08.900 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:08.929 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:09.009 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:33:09.117 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 735 kB/s | 248 kB 00:00 2024-03-29T11:33:09.292 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-29T11:33:09.376 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-03-29T11:33:09.401 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 531 kB/s | 58 kB 00:00 2024-03-29T11:33:09.426 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-29T11:33:09.560 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 177 kB/s | 138 kB 00:00 2024-03-29T11:33:09.562 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:09.562 INFO:teuthology.orchestra.run.smithi183.stdout:Total 7.5 MB/s | 6.8 MB 00:00 2024-03-29T11:33:09.575 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:33:09.584 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:33:09.584 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:33:09.668 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:33:09.672 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:33:09.672 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:33:09.784 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:33:09.968 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:33:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:10.693 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:10.865 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:11.024 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:33:11.664 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:11.664 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:11.664 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:11.664 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:11.664 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:11.664 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:11.992 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:11.992 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:11.992 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:33:11.992 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:11.992 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:11.992 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:11.992 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:11.992 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:11.992 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:11.993 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:11.993 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:12.065 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:33:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:12.255 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse 2024-03-29T11:33:12.825 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:05 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:12.974 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:12.974 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:12.975 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T11:33:12.975 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:12.975 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:33:12.975 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:33:12.975 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T11:33:12.975 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:33:12.975 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:12.975 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:33:12.975 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:12.975 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-29T11:33:12.975 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:12.976 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 964 k 2024-03-29T11:33:12.976 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.8 M 2024-03-29T11:33:12.976 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:33:13.112 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 579 kB/s | 79 kB 00:00 2024-03-29T11:33:13.471 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.7 MB/s | 885 kB 00:00 2024-03-29T11:33:13.472 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:13.472 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.9 MB/s | 964 kB 00:00 2024-03-29T11:33:13.473 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:33:13.493 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:33:13.493 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:33:13.567 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:33:13.567 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:33:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:33:13.953 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:14.628 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:14.628 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:33:14.759 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:14.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:14.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:14.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:14.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:14.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:14.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:14.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:14.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:14.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:14.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:14.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:14.948 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:33:14.948 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:14.948 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:33:14.948 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:33:14.949 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:14.949 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:15.155 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:15.157 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:15.157 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:15.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-volume 2024-03-29T11:33:15.520 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:15.850 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:08 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:16.003 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:16.003 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:16.003 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T11:33:16.003 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:16.003 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:33:16.003 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:33:16.003 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T11:33:16.004 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:33:16.004 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:16.004 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:33:16.004 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:16.004 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-29T11:33:16.004 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:16.004 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 332 k 2024-03-29T11:33:16.005 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.5 M 2024-03-29T11:33:16.005 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:33:16.091 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:16.134 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 532 kB/s | 69 kB 00:00 2024-03-29T11:33:16.249 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:16.250 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T11:33:16.250 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T11:33:16.250 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T11:33:16.250 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:33:16.250 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:16.250 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T11:33:16.250 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:16.250 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T11:33:16.251 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2024-03-29T11:33:16.252 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:16.252 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.0 M 2024-03-29T11:33:16.252 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2024-03-29T11:33:16.252 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:33:16.376 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 711 kB/s | 264 kB 00:00 2024-03-29T11:33:16.377 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:16.377 INFO:teuthology.orchestra.run.smithi183.stdout:Total 890 kB/s | 332 kB 00:00 2024-03-29T11:33:16.377 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:33:16.386 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:33:16.386 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:33:16.428 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:33:16.428 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:33:16.524 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 199 kB/s | 24 kB 00:00 2024-03-29T11:33:16.549 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 122 kB/s | 18 kB 00:00 2024-03-29T11:33:16.577 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:33:16.599 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:33:16.633 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-29T11:33:16.658 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-29T11:33:16.683 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-29T11:33:16.709 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 170 kB/s | 52 kB 00:00 2024-03-29T11:33:16.766 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:33:16.803 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:16.820 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:16.820 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:16.820 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:33:16.820 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:16.934 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-03-29T11:33:16.959 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-03-29T11:33:17.110 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 395 kB/s | 59 kB 00:00 2024-03-29T11:33:17.135 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 476 kB/s | 203 kB 00:00 2024-03-29T11:33:17.227 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 768 kB/s | 90 kB 00:00 2024-03-29T11:33:17.269 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-29T11:33:17.273 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:17.273 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-29T11:33:17.273 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:33:17.286 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:33:17.286 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:33:17.377 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:33:17.377 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:33:17.679 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:33:17.888 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:33:18.013 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:33:18.013 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:18.013 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:33:18.013 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:18.013 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:33:18.013 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:18.013 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:18.039 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:18.094 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:18.149 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:18.171 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:18.212 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:18.306 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:18.391 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel 2024-03-29T11:33:18.415 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:18.472 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:18.971 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:11 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:19.122 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 127 k 2024-03-29T11:33:19.123 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 456 k 2024-03-29T11:33:19.124 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:33:19.194 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:19.244 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:19.278 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:19.326 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:19.348 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 563 kB/s | 127 kB 00:00 2024-03-29T11:33:19.349 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:19.349 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:19.349 INFO:teuthology.orchestra.run.smithi183.stdout:Total 560 kB/s | 127 kB 00:00 2024-03-29T11:33:19.349 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:33:19.355 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:33:19.356 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:33:19.411 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:33:19.412 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:33:19.558 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:33:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:20.139 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:20.153 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:20.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:20.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:20.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:20.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:20.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:20.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:20.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:20.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:20.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:20.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:20.154 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:20.154 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:20.437 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:20.437 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:20.437 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:33:20.438 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:20.438 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:20.438 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:20.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:20.533 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:20.533 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:33:20.533 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:20.533 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:20.533 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:20.534 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:20.534 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:20.534 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:20.534 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:20.535 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:20.535 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:20.535 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:20.535 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:20.535 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:20.535 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:20.535 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:20.535 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:20.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2 2024-03-29T11:33:20.842 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:21.277 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:21.387 INFO:teuthology.orchestra.run.smithi183.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:21.957 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:21.957 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:21.957 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T11:33:21.957 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T11:33:21.957 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T11:33:21.957 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:33:21.957 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:21.957 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 6.8 M 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 33 M 2024-03-29T11:33:21.958 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:33:21.959 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:21.959 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T11:33:21.959 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:21.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs-devel 2024-03-29T11:33:22.064 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-29T11:33:22.130 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 593 kB/s | 138 kB 00:00 2024-03-29T11:33:22.248 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-29T11:33:22.473 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 169 kB/s | 58 kB 00:00 2024-03-29T11:33:22.532 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:22.539 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 191 kB/s | 51 kB 00:00 2024-03-29T11:33:22.565 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:33:22.568 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:22.569 INFO:teuthology.orchestra.run.smithi155.stdout:Total 6.8 MB/s | 6.8 MB 00:00 2024-03-29T11:33:22.582 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:33:22.593 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:33:22.593 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:33:22.683 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:33:22.684 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:22.685 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:22.685 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-29T11:33:22.685 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:22.685 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:33:22.685 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:33:22.685 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:22.685 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:33:22.685 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:22.685 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T11:33:22.686 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:22.686 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 31 k 2024-03-29T11:33:22.686 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 140 k 2024-03-29T11:33:22.686 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:33:22.686 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:33:22.852 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 187 kB/s | 31 kB 00:00 2024-03-29T11:33:22.853 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:22.853 INFO:teuthology.orchestra.run.smithi183.stdout:Total 185 kB/s | 31 kB 00:00 2024-03-29T11:33:22.853 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:33:22.857 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:33:22.857 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:33:22.884 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:33:22.884 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:33:22.987 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:33:23.019 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:33:23.109 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:23.158 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:23.200 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:23.458 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:23.726 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:23.736 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:23.736 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:23.736 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:33:23.736 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:23.736 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:23.736 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:23.812 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:23.873 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:23.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2 2024-03-29T11:33:23.904 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:24.453 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:24.550 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:24.602 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:24.603 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T11:33:24.603 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:24.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1 2024-03-29T11:33:24.700 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:24.700 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:24.700 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:24.701 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:24.701 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:24.701 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:25.006 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:25.007 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:25.007 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:33:25.007 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:25.007 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:25.007 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:25.007 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:25.007 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:25.007 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:25.007 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:25.007 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:25.228 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:25.267 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse 2024-03-29T11:33:25.325 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:25.376 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:25.377 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T11:33:25.377 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:25.430 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados 2024-03-29T11:33:25.835 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:25.985 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:25.986 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:25.986 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T11:33:25.986 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:25.986 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:33:25.986 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:33:25.986 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T11:33:25.987 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:33:25.987 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:25.987 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:33:25.987 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:25.987 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-03-29T11:33:25.987 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:25.987 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 964 k 2024-03-29T11:33:25.987 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.8 M 2024-03-29T11:33:25.987 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:33:25.997 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:26.095 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:26.101 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 693 kB/s | 79 kB 00:00 2024-03-29T11:33:26.148 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:26.149 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T11:33:26.149 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:26.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw 2024-03-29T11:33:26.393 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.1 MB/s | 885 kB 00:00 2024-03-29T11:33:26.394 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:26.394 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.3 MB/s | 964 kB 00:00 2024-03-29T11:33:26.395 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:33:26.414 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:33:26.414 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:33:26.488 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:33:26.488 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:33:26.676 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:33:26.774 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:26.803 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:33:26.871 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:26.875 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:26.923 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:26.924 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T11:33:26.924 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:26.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs 2024-03-29T11:33:27.550 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:27.569 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:27.569 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:33:27.648 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:27.699 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:27.700 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T11:33:27.700 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:27.752 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd 2024-03-29T11:33:27.901 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:33:27.901 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:27.901 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:33:27.901 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:33:27.901 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:27.901 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:28.192 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-volume 2024-03-29T11:33:28.317 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:28.416 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:28.468 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:28.469 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T11:33:28.469 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:28.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse 2024-03-29T11:33:28.765 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:28.922 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:28.924 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T11:33:28.925 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:28.925 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:33:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:33:28.925 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T11:33:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:33:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:28.926 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:33:28.926 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:28.926 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-03-29T11:33:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:28.927 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 332 k 2024-03-29T11:33:28.927 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.5 M 2024-03-29T11:33:28.927 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:33:29.056 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 521 kB/s | 69 kB 00:00 2024-03-29T11:33:29.126 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:29.275 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:29.275 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:29.275 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:29.275 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:29.275 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:33:29.276 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:33:29.276 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:29.276 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:33:29.276 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:29.276 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T11:33:29.276 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:29.277 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 85 k 2024-03-29T11:33:29.277 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 231 k 2024-03-29T11:33:29.277 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:33:29.290 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 722 kB/s | 264 kB 00:00 2024-03-29T11:33:29.291 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:29.291 INFO:teuthology.orchestra.run.smithi155.stdout:Total 903 kB/s | 332 kB 00:00 2024-03-29T11:33:29.291 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:33:29.300 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:33:29.300 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:33:29.336 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:33:29.336 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:33:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:33:29.488 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 402 kB/s | 85 kB 00:00 2024-03-29T11:33:29.489 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:29.489 INFO:teuthology.orchestra.run.smithi183.stdout:Total 400 kB/s | 85 kB 00:00 2024-03-29T11:33:29.489 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:33:29.496 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:33:29.496 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:33:29.551 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:33:29.552 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:33:29.665 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:33:29.685 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:33:29.703 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:29.718 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:29.719 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:29.719 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:33:29.719 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:29.787 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:30.203 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:30.399 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:33:30.520 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:30.520 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:30.520 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:33:30.520 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:30.520 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:30.520 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:30.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror 2024-03-29T11:33:30.755 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:33:30.756 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:30.756 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:33:30.756 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:30.756 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:33:30.756 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:30.756 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:31.121 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel 2024-03-29T11:33:31.324 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:31.478 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:31.479 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:31.479 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-29T11:33:31.479 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:31.479 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:33:31.480 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:33:31.480 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:31.480 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:33:31.480 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:31.480 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T11:33:31.480 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:31.480 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 3.1 M 2024-03-29T11:33:31.480 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 13 M 2024-03-29T11:33:31.480 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:33:31.751 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:31.906 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:31.906 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:31.906 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-29T11:33:31.907 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:31.907 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:33:31.907 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:33:31.907 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:31.907 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:33:31.907 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:31.907 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T11:33:31.907 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:31.907 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 127 k 2024-03-29T11:33:31.907 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 456 k 2024-03-29T11:33:31.907 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:33:31.942 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 6.7 MB/s | 3.1 MB 00:00 2024-03-29T11:33:31.943 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:31.943 INFO:teuthology.orchestra.run.smithi183.stdout:Total 6.7 MB/s | 3.1 MB 00:00 2024-03-29T11:33:31.944 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:33:31.953 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:33:31.954 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:33:32.028 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:33:32.028 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:33:32.240 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 381 kB/s | 127 kB 00:00 2024-03-29T11:33:32.240 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:32.241 INFO:teuthology.orchestra.run.smithi155.stdout:Total 379 kB/s | 127 kB 00:00 2024-03-29T11:33:32.241 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:33:32.246 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:33:32.246 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:33:32.302 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:33:32.302 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:33:32.448 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:33:32.462 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:33:32.509 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:32.534 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:32.535 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:32.535 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:33:32.535 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:32.535 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:32.535 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:32.548 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:32.996 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:33.297 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:33.298 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:33.298 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:33:33.298 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:33.298 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:33.298 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:33.554 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2 2024-03-29T11:33:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:33.680 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:33:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:33.680 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:33.931 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd 2024-03-29T11:33:34.133 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:34.237 INFO:teuthology.orchestra.run.smithi155.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:34.292 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:34.293 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T11:33:34.293 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:34.365 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs-devel 2024-03-29T11:33:34.508 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:31:07 AM UTC. 2024-03-29T11:33:34.656 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T11:33:34.657 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:34.657 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:34.657 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:34.657 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T11:33:34.657 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:33:34.657 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:34.658 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T11:33:34.658 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T11:33:34.658 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T11:33:34.658 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:34.658 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 171 k 2024-03-29T11:33:34.658 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 494 k 2024-03-29T11:33:34.658 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T11:33:34.937 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:34.978 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 535 kB/s | 171 kB 00:00 2024-03-29T11:33:34.979 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:34.979 INFO:teuthology.orchestra.run.smithi183.stdout:Total 533 kB/s | 171 kB 00:00 2024-03-29T11:33:34.979 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T11:33:34.987 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T11:33:34.987 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T11:33:35.043 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T11:33:35.044 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T11:33:35.089 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:35.090 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:35.090 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-29T11:33:35.090 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:35.090 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:33:35.090 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:33:35.090 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:35.091 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:33:35.091 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:35.091 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T11:33:35.091 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:35.091 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 31 k 2024-03-29T11:33:35.091 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 140 k 2024-03-29T11:33:35.091 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:33:35.194 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T11:33:35.260 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 185 kB/s | 31 kB 00:00 2024-03-29T11:33:35.261 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:35.261 INFO:teuthology.orchestra.run.smithi155.stdout:Total 183 kB/s | 31 kB 00:00 2024-03-29T11:33:35.261 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:33:35.267 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:33:35.267 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:33:35.291 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:33:35.292 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:33:35.302 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:33:35.518 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:36.003 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:36.003 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:36.003 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T11:33:36.003 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:36.003 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:36.003 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T11:33:36.171 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:36.171 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:36.172 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:33:36.172 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:36.172 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:36.172 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:36.236 DEBUG:teuthology.parallel:result is None 2024-03-29T11:33:36.324 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2 2024-03-29T11:33:36.883 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:36.980 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:37.031 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:37.032 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T11:33:37.032 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:37.078 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1 2024-03-29T11:33:37.641 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:37.738 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:37.790 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:37.791 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T11:33:37.791 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:37.837 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados 2024-03-29T11:33:38.401 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:38.499 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:38.551 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:38.553 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T11:33:38.553 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:38.599 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw 2024-03-29T11:33:39.169 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:39.267 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:39.319 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:39.320 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T11:33:39.320 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:39.366 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs 2024-03-29T11:33:39.941 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:40.039 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:40.091 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:40.092 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T11:33:40.092 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:40.133 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd 2024-03-29T11:33:40.703 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:34 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:40.802 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:40.854 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:40.855 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T11:33:40.855 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:40.904 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse 2024-03-29T11:33:41.470 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:41.620 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:41.621 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:41.621 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:41.621 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:41.621 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:33:41.621 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:33:41.621 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:41.621 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:33:41.621 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:41.621 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T11:33:41.622 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:41.622 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 85 k 2024-03-29T11:33:41.622 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 231 k 2024-03-29T11:33:41.622 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:33:41.971 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 243 kB/s | 85 kB 00:00 2024-03-29T11:33:41.972 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:41.972 INFO:teuthology.orchestra.run.smithi155.stdout:Total 243 kB/s | 85 kB 00:00 2024-03-29T11:33:41.972 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:33:41.979 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:33:41.980 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:33:42.034 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:33:42.035 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:33:42.171 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:33:42.279 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:42.690 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:42.998 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:42.998 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:42.998 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:33:42.998 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:42.998 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:42.998 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:43.234 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror 2024-03-29T11:33:43.832 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:37 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:43.991 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.1 M 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 13 M 2024-03-29T11:33:43.992 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:33:44.663 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 4.6 MB/s | 3.1 MB 00:00 2024-03-29T11:33:44.663 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:44.663 INFO:teuthology.orchestra.run.smithi155.stdout:Total 4.6 MB/s | 3.1 MB 00:00 2024-03-29T11:33:44.663 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:33:44.670 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:33:44.670 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:33:44.742 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:33:44.743 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:33:45.170 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:33:45.235 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:45.261 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:45.261 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:45.261 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:33:45.261 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:45.261 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:45.261 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:46.397 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:46.398 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:46.398 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:33:46.398 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:46.398 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:46.398 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:46.629 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd 2024-03-29T11:33:47.211 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:41 ago on Fri 29 Mar 2024 11:31:06 AM UTC. 2024-03-29T11:33:47.359 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T11:33:47.360 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:47.360 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:47.360 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:47.360 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T11:33:47.360 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:33:47.360 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:47.361 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T11:33:47.361 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T11:33:47.361 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T11:33:47.361 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:47.361 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 171 k 2024-03-29T11:33:47.361 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 494 k 2024-03-29T11:33:47.361 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T11:33:47.589 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 754 kB/s | 171 kB 00:00 2024-03-29T11:33:47.590 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:47.590 INFO:teuthology.orchestra.run.smithi155.stdout:Total 749 kB/s | 171 kB 00:00 2024-03-29T11:33:47.590 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T11:33:47.597 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T11:33:47.597 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T11:33:47.655 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T11:33:47.656 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T11:33:47.802 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T11:33:47.921 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:48.363 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:48.655 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:48.655 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:48.656 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T11:33:48.656 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:48.656 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:48.656 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T11:33:48.877 DEBUG:teuthology.parallel:result is None 2024-03-29T11:33:48.878 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:33:49.102 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:33:49.123 INFO:teuthology.orchestra.run.smithi155.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:49.123 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:49.123 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:33:49.125 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:33:49.351 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:33:49.376 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:49.377 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:49.377 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:33:49.379 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:33:49.379 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:33:49.379 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:33:49.413 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:33:49.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:33:49.448 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:33:49.449 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:33:49.449 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:33:49.480 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:33:49.548 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:33:49.548 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:33:49.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:33:49.650 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:33:49.650 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:33:49.650 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:33:49.681 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:33:49.752 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:33:49.752 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:33:49.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:33:49.853 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:33:49.853 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:33:49.853 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:33:49.882 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:33:49.952 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:33:49.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:33:49.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:33:50.052 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:33:51.302 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:33:51.302 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /var/log/ceph 2024-03-29T11:33:51.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /var/log/ceph 2024-03-29T11:33:51.336 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:33:51.336 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:33:51.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:33:51.409 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:33:51.409 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:33:51.455 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:33:51.641 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:33:51.641 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:33:51.641 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622710', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi155.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHV5Cxb2pgYCJ71n+4eseDt81EMdVU/jdelCc8I0IKUkVzbtbHdsAdlkS3a1HSHERpFUUwZ5c9F6VwCnfo9B8uU=', 'smithi183.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGlI3AOi5gl6ENZptPZ3nn1/HIM53rjn8vZ9KNQE9ERnA1k0FrtNBNenc01NO2wr+dWLmdyahceIHz3EWoFGyI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:33:51.642 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:33:51.653 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:33:51.669 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:33:51.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:33:51.743 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:33:51.743 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:33:51.801 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:33:51.802 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-03-29T11:33:51.860 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:33:51.860 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:51.860 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:33:51.860 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:51.860 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:51.860 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-29 11:33:45.508307874 +0000 2024-03-29T11:33:51.860 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-29 11:30:19.688224465 +0000 2024-03-29T11:33:51.860 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-29 11:30:19.688224465 +0000 2024-03-29T11:33:51.860 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-29 11:30:19.688224465 +0000 2024-03-29T11:33:51.861 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:33:51.928 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-29T11:33:51.928 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-29T11:33:51.928 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.00024807 s, 2.1 MB/s 2024-03-29T11:33:51.929 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:33:51.988 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-03-29T11:33:52.044 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:33:52.045 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:52.045 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:33:52.045 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:52.045 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:52.046 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-29 11:33:45.509307855 +0000 2024-03-29T11:33:52.046 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-29 11:30:20.031217790 +0000 2024-03-29T11:33:52.046 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-29 11:30:20.031217790 +0000 2024-03-29T11:33:52.046 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-29 11:30:20.031217790 +0000 2024-03-29T11:33:52.046 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:33:52.113 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-29T11:33:52.113 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-29T11:33:52.113 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.00017433 s, 2.9 MB/s 2024-03-29T11:33:52.114 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:33:52.172 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-03-29T11:33:52.228 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:33:52.228 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:52.228 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:33:52.228 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:52.228 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:52.228 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-29 11:33:45.509307855 +0000 2024-03-29T11:33:52.228 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-29 11:30:20.374211115 +0000 2024-03-29T11:33:52.228 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-29 11:30:20.374211115 +0000 2024-03-29T11:33:52.229 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-29 11:30:20.374211115 +0000 2024-03-29T11:33:52.229 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:33:52.294 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-29T11:33:52.295 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-29T11:33:52.295 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000212487 s, 2.4 MB/s 2024-03-29T11:33:52.296 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:33:52.352 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-03-29T11:33:52.407 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:33:52.407 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:52.407 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:33:52.407 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:52.407 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:52.407 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-29 11:33:45.509307855 +0000 2024-03-29T11:33:52.407 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-29 11:30:20.722204343 +0000 2024-03-29T11:33:52.407 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-29 11:30:20.722204343 +0000 2024-03-29T11:33:52.407 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-29 11:30:20.722204343 +0000 2024-03-29T11:33:52.408 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:33:52.474 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-29T11:33:52.474 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-29T11:33:52.474 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000212958 s, 2.4 MB/s 2024-03-29T11:33:52.475 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:33:52.531 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:52.531 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:33:52.531 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:33:52.548 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:33:52.548 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-03-29T11:33:52.604 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:33:52.604 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:52.604 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:33:52.604 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:52.604 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:52.604 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-29 11:33:32.800261372 +0000 2024-03-29T11:33:52.604 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-29 11:30:20.640366906 +0000 2024-03-29T11:33:52.605 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-29 11:30:20.640366906 +0000 2024-03-29T11:33:52.605 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-29 11:30:20.640366906 +0000 2024-03-29T11:33:52.605 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:33:52.671 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-29T11:33:52.671 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-29T11:33:52.672 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000267912 s, 1.9 MB/s 2024-03-29T11:33:52.673 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:33:52.730 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-03-29T11:33:52.786 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:33:52.786 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:52.786 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:33:52.786 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:52.786 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:52.786 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-29 11:33:32.801261346 +0000 2024-03-29T11:33:52.786 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-29 11:30:20.989357935 +0000 2024-03-29T11:33:52.786 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-29 11:30:20.989357935 +0000 2024-03-29T11:33:52.786 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-29 11:30:20.989357935 +0000 2024-03-29T11:33:52.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:33:52.853 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-29T11:33:52.853 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-29T11:33:52.853 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000211935 s, 2.4 MB/s 2024-03-29T11:33:52.854 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:33:52.910 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-03-29T11:33:52.965 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:33:52.965 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:52.965 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T11:33:52.965 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:52.965 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:52.965 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-29 11:33:32.801261346 +0000 2024-03-29T11:33:52.965 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-29 11:30:21.323349349 +0000 2024-03-29T11:33:52.965 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-29 11:30:21.323349349 +0000 2024-03-29T11:33:52.965 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-29 11:30:21.323349349 +0000 2024-03-29T11:33:52.965 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:33:53.030 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-29T11:33:53.031 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-29T11:33:53.031 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000233276 s, 2.2 MB/s 2024-03-29T11:33:53.032 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:33:53.087 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-03-29T11:33:53.142 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:33:53.142 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:53.142 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:33:53.143 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:53.143 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:53.143 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-29 11:33:32.801261346 +0000 2024-03-29T11:33:53.143 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-29 11:30:21.679340198 +0000 2024-03-29T11:33:53.143 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-29 11:30:21.679340198 +0000 2024-03-29T11:33:53.143 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-29 11:30:21.679340198 +0000 2024-03-29T11:33:53.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:33:53.208 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-29T11:33:53.208 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-29T11:33:53.208 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000209094 s, 2.4 MB/s 2024-03-29T11:33:53.209 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:33:53.265 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:53.266 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:33:53.266 INFO:tasks.ceph:Generating config... 2024-03-29T11:33:53.267 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:33:53.267 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:33:53.267 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:33:53.267 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:33:53.267 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:33:53.267 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:33:53.267 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:33:53.267 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:33:53.268 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:33:53.268 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:33:53.268 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:33:53.269 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:33:53.269 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:33:53.269 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:33:53.269 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:33:53.269 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:33:53.269 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:33:53.269 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:33:53.269 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:33:53.269 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:33:53.270 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:33:53.271 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:33:53.271 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:33:53.271 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:33:53.271 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:33:53.271 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:33:53.271 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:33:53.271 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:33:53.326 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:33:53.329 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:33:53.375 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:33:53.402 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.155:6789]'), ('mon.c', '[v1:172.21.15.155:6790]'), ('mon.b', '[v1:172.21.15.183:6789]')] 2024-03-29T11:33:53.402 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.155:6789],[v1:172.21.15.155:6790],[v1:172.21.15.183:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:33:53.403 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:33:53.403 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:33:53.457 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.155:6789]' --addv c '[v1:172.21.15.155:6790]' --addv b '[v1:172.21.15.183:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:33:53.542 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:33:53.542 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: generated fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:33:53.542 INFO:teuthology.orchestra.run.smithi155.stdout:setting min_mon_release = quincy 2024-03-29T11:33:53.542 INFO:teuthology.orchestra.run.smithi155.stdout:epoch 0 2024-03-29T11:33:53.542 INFO:teuthology.orchestra.run.smithi155.stdout:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:33:53.542 INFO:teuthology.orchestra.run.smithi155.stdout:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:33:53.542 INFO:teuthology.orchestra.run.smithi155.stdout:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:33:53.542 INFO:teuthology.orchestra.run.smithi155.stdout:min_mon_release 17 (quincy) 2024-03-29T11:33:53.542 INFO:teuthology.orchestra.run.smithi155.stdout:election_strategy: 1 2024-03-29T11:33:53.542 INFO:teuthology.orchestra.run.smithi155.stdout:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:33:53.543 INFO:teuthology.orchestra.run.smithi155.stdout:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:33:53.543 INFO:teuthology.orchestra.run.smithi155.stdout:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:33:53.543 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:33:53.544 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:33:53.558 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2... 2024-03-29T11:33:53.559 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:33:53.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:33:53.648 INFO:teuthology.orchestra.run.smithi155.stdout:[global] 2024-03-29T11:33:53.648 INFO:teuthology.orchestra.run.smithi155.stdout: chdir = "" 2024-03-29T11:33:53.648 INFO:teuthology.orchestra.run.smithi155.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:33:53.648 INFO:teuthology.orchestra.run.smithi155.stdout: auth supported = cephx 2024-03-29T11:33:53.648 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.648 INFO:teuthology.orchestra.run.smithi155.stdout: filestore xattr use omap = true 2024-03-29T11:33:53.648 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.648 INFO:teuthology.orchestra.run.smithi155.stdout: mon clock drift allowed = 1.000 2024-03-29T11:33:53.648 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: osd crush chooseleaf type = 0 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: auth debug = true 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on old message = true 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on bug = true 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: mon pg warn max object skew = 0 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.649 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default size = 2 2024-03-29T11:33:53.650 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.650 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow primary affinity = true 2024-03-29T11:33:53.650 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow pg remap = true 2024-03-29T11:33:53.650 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:33:53.650 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:33:53.650 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on no sortbitwise = false 2024-03-29T11:33:53.650 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:33:53.650 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on too few osds = false 2024-03-29T11:33:53.650 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:33:53.650 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:33:53.651 INFO:teuthology.orchestra.run.smithi155.stdout: mon_allow_pool_size_one = true 2024-03-29T11:33:53.651 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.651 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:33:53.651 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.651 INFO:teuthology.orchestra.run.smithi155.stdout: osd default data pool replay window = 5 2024-03-29T11:33:53.651 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.651 INFO:teuthology.orchestra.run.smithi155.stdout: mon allow pool delete = true 2024-03-29T11:33:53.651 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.651 INFO:teuthology.orchestra.run.smithi155.stdout: mon cluster log file level = debug 2024-03-29T11:33:53.651 INFO:teuthology.orchestra.run.smithi155.stdout: debug asserts on shutdown = true 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: mon health detail to clog = false 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: mon host = "[v1:172.21.15.155:6789],[v1:172.21.15.155:6790],[v1:172.21.15.183:6789]" 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: lockdep = True 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: mon client directed command retry = 5 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: mon election default strategy = 1 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: ms bind msgr2 = False 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: ms inject socket failures = 5000 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: ms type = async 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: fsid = 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout:[osd] 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: osd journal size = 100 2024-03-29T11:33:53.652 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub load threshold = 5.0 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub max interval = 600 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: osd recover clone overlap = true 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: osd recovery max chunk = 1048576 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug shutdown = true 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug op order = true 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify stray on activate = true 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug trim objects = true 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: osd open classes on start = true 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug pg log writeout = true 2024-03-29T11:33:53.653 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: osd map max advance = 10 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: journal zero on create = true 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: filestore apply finisher threads = 3 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: bdev debug aio = true 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug misdirected ops = true 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore block size = 96636764160 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore compression algorithm = snappy 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore compression mode = aggressive 2024-03-29T11:33:53.654 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore fsck on mount = True 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore zero block detection = True 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluefs = 20 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluestore = 20 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: debug monc = 20 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: debug osd = 20 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: debug rocksdb = 10 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd full ratio = 0.9 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: osd class default list = * 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: osd class load list = * 2024-03-29T11:33:53.655 INFO:teuthology.orchestra.run.smithi155.stdout: osd client watch timeout = 120 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify cached snaps = True 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify missing on start = True 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock override recovery settings = True 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: osd objectstore = bluestore 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue = debug_random 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue cut off = debug_random 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: osd shutdown pgref assert = True 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr] 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: debug mgr = 20 2024-03-29T11:33:53.656 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: mgr/telemetry/nag = false 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout:[mon] 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: debug paxos = 20 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: mon data avail warn = 5 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: mon mgr mkfs grace = 240 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd reporter subtree level = osd 2024-03-29T11:33:53.657 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd prime pg temp = true 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: # 1m isn't quite enough 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:33:53.658 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: mon min osdmap epochs = 50 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune interval = 2 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune min = 15 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: mon scrub interval = 300 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on pool no app = False 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: paxos service trim min = 10 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout:[client] 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: rgw cache enabled = true 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable ops log = true 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable usage log = true 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:33:53.659 INFO:teuthology.orchestra.run.smithi155.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:33:53.660 INFO:teuthology.orchestra.run.smithi155.stdout: debug monc = 20 2024-03-29T11:33:53.660 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-29T11:33:53.660 INFO:teuthology.orchestra.run.smithi155.stdout: debug objecter = 20 2024-03-29T11:33:53.660 INFO:teuthology.orchestra.run.smithi155.stdout: debug rados = 20 2024-03-29T11:33:53.660 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.a] 2024-03-29T11:33:53.660 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.c] 2024-03-29T11:33:53.660 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.b] 2024-03-29T11:33:53.660 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-29T11:33:53.661 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-29T11:33:53.661 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:33:53.661 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-29T11:33:53.661 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.661 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-29T11:33:53.661 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.661 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-29T11:33:53.661 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.661 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-29T11:33:53.661 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.662 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.663 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "[v1:172.21.15.155:6789],[v1:172.21.15.155:6790],[v1:172.21.15.183:6789]" 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: lockdep = True 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: mon client directed command retry = 5 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: mon election default strategy = 1 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: ms bind msgr2 = False 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: ms inject socket failures = 5000 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: ms type = async 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-29T11:33:53.664 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-29T11:33:53.665 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.665 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-29T11:33:53.665 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-29T11:33:53.665 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:33:53.665 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.665 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-29T11:33:53.665 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-29T11:33:53.665 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.665 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-29T11:33:53.665 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-29T11:33:53.665 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-29T11:33:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore compression algorithm = snappy 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore compression mode = aggressive 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore zero block detection = True 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 20 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 20 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 20 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 10 2024-03-29T11:33:53.667 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: osd class default list = * 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: osd class load list = * 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: osd client watch timeout = 120 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify cached snaps = True 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify missing on start = True 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock override recovery settings = True 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: osd op queue = debug_random 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: osd op queue cut off = debug_random 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: osd shutdown pgref assert = True 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.668 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-29T11:33:53.669 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:33:53.670 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.671 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-29T11:33:53.671 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:33:53.671 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.671 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:33:53.671 INFO:teuthology.orchestra.run.smithi183.stdout: mon min osdmap epochs = 50 2024-03-29T11:33:53.671 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune interval = 2 2024-03-29T11:33:53.671 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune min = 15 2024-03-29T11:33:53.671 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:33:53.671 INFO:teuthology.orchestra.run.smithi183.stdout: mon scrub interval = 300 2024-03-29T11:33:53.671 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on pool no app = False 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout: paxos service trim min = 10 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 20 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout: debug objecter = 20 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout: debug rados = 20 2024-03-29T11:33:53.672 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-29T11:33:53.673 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-03-29T11:33:53.673 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-03-29T11:33:53.674 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:33:53.674 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:33:53.769 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:33:53.770 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:33:53.770 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:33:53.785 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:33:53.785 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:33:53.839 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:33:53.840 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:33:53.840 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:33:53.840 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:33:53.922 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:33:53.922 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:33:53.977 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:33:53.977 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:33:53.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:33:53.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:33:54.017 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:33:54.017 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:33:54.071 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:33:54.072 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:33:54.072 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:33:54.127 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:33:54.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:33:54.201 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:33:54.204 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:33:54.204 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:33:54.204 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:33:54.248 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:33:54.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:33:54.305 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:33:54.320 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:33:54.321 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:33:54.321 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:33:54.348 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:54.348 INFO:tasks.ceph:role: osd.0 2024-03-29T11:33:54.348 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:33:54.348 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:33:54.422 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:33:54.422 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:33:54.422 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:33:54.422 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:33:54.422 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:33:54.422 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:33:54.422 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:33:54.422 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:33:54.423 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:33:54.423 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:33:54.929 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-29T11:33:54.931 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-29T11:33:54.932 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:33:55.075 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:33:55.110 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:33:55.177 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:55.178 INFO:tasks.ceph:role: osd.1 2024-03-29T11:33:55.178 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:33:55.178 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:33:55.252 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:33:55.252 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:33:55.252 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:33:55.252 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:33:55.252 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:33:55.252 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:33:55.252 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:33:55.252 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:33:55.253 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:33:55.253 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:33:55.694 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-29T11:33:55.696 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-29T11:33:55.696 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:33:55.788 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:33:55.819 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:33:55.886 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:55.886 INFO:tasks.ceph:role: osd.2 2024-03-29T11:33:55.886 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:33:55.886 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:33:55.960 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:33:55.960 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:33:55.960 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:33:55.960 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:33:55.960 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:33:55.960 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:33:55.960 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:33:55.960 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:33:55.960 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:33:55.960 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:33:56.441 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-29T11:33:56.443 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-29T11:33:56.444 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:33:56.596 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:33:56.627 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:33:56.694 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:56.694 INFO:tasks.ceph:role: osd.3 2024-03-29T11:33:56.694 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:33:56.694 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:33:56.768 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:33:56.768 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:33:56.768 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:33:56.768 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:33:56.768 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:33:56.768 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:33:56.768 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:33:56.769 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:33:56.769 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:33:56.769 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:33:57.300 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-29T11:33:57.302 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-29T11:33:57.302 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:33:57.441 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:33:57.473 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:33:57.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:33:57.575+0000 7f9380010640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:33:57.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:33:57.576+0000 7f9380010640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:33:57.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:33:57.576+0000 7f9380010640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:33:57.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:33:57.576+0000 7f9380010640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:33:57.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:33:57.576+0000 7f9380010640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:33:57.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:33:57.576+0000 7f9380010640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:34:00.425 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:00.456 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:00.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:00.550+0000 7fe224f0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:34:00.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:00.550+0000 7fe224f0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:34:00.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:00.550+0000 7fe224f0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:00.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:00.550+0000 7fe224f0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:00.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:00.550+0000 7fe224f0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:00.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:00.551+0000 7fe224f0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:34:03.390 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:03.421 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:03.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:03.514+0000 7f8d2bc13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:34:03.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:03.515+0000 7f8d2bc13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:34:03.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:03.515+0000 7f8d2bc13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:03.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:03.515+0000 7f8d2bc13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:03.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:03.515+0000 7f8d2bc13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:03.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:03.515+0000 7f8d2bc13640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:34:06.361 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:06.399 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:06.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:06.520+0000 7fcef8eb5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:34:06.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:06.520+0000 7fcef8eb5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:34:06.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:06.520+0000 7fcef8eb5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:06.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:06.521+0000 7fcef8eb5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:06.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:06.521+0000 7fcef8eb5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:06.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:06.521+0000 7fcef8eb5640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:34:09.372 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:09.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:09.437 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:09.437 INFO:tasks.ceph:role: osd.4 2024-03-29T11:34:09.437 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:34:09.437 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:34:09.514 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:09.515 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:09.515 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:09.515 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:09.515 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:09.515 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:09.515 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:09.516 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:09.516 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:09.516 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:10.029 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-29T11:34:10.032 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-29T11:34:10.032 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:10.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:10.195 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:10.265 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:10.265 INFO:tasks.ceph:role: osd.5 2024-03-29T11:34:10.266 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:34:10.266 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:34:10.343 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:10.343 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:10.343 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:10.343 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:10.343 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:10.343 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:10.343 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:10.343 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:10.343 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:10.343 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:10.784 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-29T11:34:10.786 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-29T11:34:10.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:10.881 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:10.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:10.979 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:10.980 INFO:tasks.ceph:role: osd.6 2024-03-29T11:34:10.980 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:34:10.980 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:11.055 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:11.055 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:11.055 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:11.055 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:11.055 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:11.055 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:11.055 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:11.538 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-29T11:34:11.540 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-29T11:34:11.540 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:11.692 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:11.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:11.792 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:11.792 INFO:tasks.ceph:role: osd.7 2024-03-29T11:34:11.792 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:34:11.792 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:34:11.866 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:11.866 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:11.867 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:11.867 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:11.867 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:12.380 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-29T11:34:12.382 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-29T11:34:12.383 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:12.525 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:12.559 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:12.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:12.666+0000 7f0067a13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:34:12.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:12.666+0000 7f0067a13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:34:12.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:12.666+0000 7f0067a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:12.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:12.666+0000 7f0067a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:12.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:12.666+0000 7f0067a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:12.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:12.666+0000 7f0067a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:34:15.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:15.546 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:15.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:15.640+0000 7f4c4882a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:34:15.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:15.640+0000 7f4c4882a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:34:15.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:15.640+0000 7f4c4882a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:15.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:15.640+0000 7f4c4882a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:15.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:15.640+0000 7f4c4882a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:15.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:15.641+0000 7f4c4882a640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:34:18.493 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:18.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:18.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:18.621+0000 7f108d010640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:34:18.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:18.621+0000 7f108d010640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:34:18.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:18.621+0000 7f108d010640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:18.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:18.621+0000 7f108d010640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:18.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:18.621+0000 7f108d010640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:18.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:18.621+0000 7f108d010640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:34:21.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:21.489 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:21.582 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:21.585+0000 7f3b5fa15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:34:21.582 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:21.585+0000 7f3b5fa15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:34:21.582 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:21.585+0000 7f3b5fa15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:21.582 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:21.585+0000 7f3b5fa15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:21.582 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:21.585+0000 7f3b5fa15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:21.582 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:21.585+0000 7f3b5fa15640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:34:24.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:24.450 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:34:24.450 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:34:24.450 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:34:24.482 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:34:24.483 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:34:24.552 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:34:24.552 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:34:24.622 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:34:24.622 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:34:24.691 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:34:24.691 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:34:24.759 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:34:24.759 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:34:24.790 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:34:24.791 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:34:24.860 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:34:24.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:34:24.929 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:34:24.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:34:24.998 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:34:24.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:34:25.066 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:34:25.066 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:34:25.083 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:34:25.083 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:34:25.122 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:34:25.122 DEBUG:teuthology.orchestra.run.smithi155:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:25.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:25.165 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.y] 2024-03-29T11:34:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAipwZm75TfBxAABOx56EYXWReGJ5F+htnOeA== 2024-03-29T11:34:25.165 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.0] 2024-03-29T11:34:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAlpwZmOFZaIhAAZI7P75QYmwfwqSZ/+atX5A== 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.1] 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAopwZmIKXTIBAALM8EfEEEpXL6Y9F1z/OOJQ== 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.2] 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQArpwZmAGzHHhAAFNI9ZDcWvAQEQ+Mgj0gJNQ== 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.3] 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAupwZme4sdHxAAdgFZPPNgwtt3Hj+80BvuIA== 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.x] 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAipwZmCdBMDBAA0R2u2FH3jNZ2AxkAYGjnGQ== 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.4] 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA0pwZmG566JxAApAp1IKvR3nf34D3y8AkOmA== 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.5] 2024-03-29T11:34:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA3pwZm0JUuJhAA/CHn89xN5f3KxXuxnc7viQ== 2024-03-29T11:34:25.167 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.6] 2024-03-29T11:34:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA6pwZmn4AKJRAAC1a2eI7bfYqqGwuD/b9abw== 2024-03-29T11:34:25.167 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.7] 2024-03-29T11:34:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA9pwZmRZboIhAAcp72KOQfGA/OM3NJ3SUSZg== 2024-03-29T11:34:25.167 INFO:teuthology.orchestra.run.smithi155.stdout:[client.0] 2024-03-29T11:34:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAipwZmiM0NDxAAdYLC4qWQIq6HxwlIU9r9Jw== 2024-03-29T11:34:25.167 INFO:teuthology.orchestra.run.smithi155.stdout:[client.1] 2024-03-29T11:34:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAipwZmNPp7EhAAkN4dQO1WWIGjo1S0BLB2bw== 2024-03-29T11:34:25.187 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.y] 2024-03-29T11:34:25.188 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAipwZm75TfBxAABOx56EYXWReGJ5F+htnOeA== 2024-03-29T11:34:25.188 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2024-03-29T11:34:25.188 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAlpwZmOFZaIhAAZI7P75QYmwfwqSZ/+atX5A== 2024-03-29T11:34:25.188 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2024-03-29T11:34:25.188 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAopwZmIKXTIBAALM8EfEEEpXL6Y9F1z/OOJQ== 2024-03-29T11:34:25.188 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2024-03-29T11:34:25.188 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQArpwZmAGzHHhAAFNI9ZDcWvAQEQ+Mgj0gJNQ== 2024-03-29T11:34:25.188 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.3] 2024-03-29T11:34:25.188 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAupwZme4sdHxAAdgFZPPNgwtt3Hj+80BvuIA== 2024-03-29T11:34:25.188 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.x] 2024-03-29T11:34:25.188 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAipwZmCdBMDBAA0R2u2FH3jNZ2AxkAYGjnGQ== 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.4] 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQA0pwZmG566JxAApAp1IKvR3nf34D3y8AkOmA== 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.5] 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQA3pwZm0JUuJhAA/CHn89xN5f3KxXuxnc7viQ== 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.6] 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQA6pwZmn4AKJRAAC1a2eI7bfYqqGwuD/b9abw== 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.7] 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQA9pwZmRZboIhAAcp72KOQfGA/OM3NJ3SUSZg== 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAipwZmiM0NDxAAdYLC4qWQIq6HxwlIU9r9Jw== 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout:[client.1] 2024-03-29T11:34:25.189 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAipwZmNPp7EhAAkN4dQO1WWIGjo1S0BLB2bw== 2024-03-29T11:34:25.190 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:25.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:25.282 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.337 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.339 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.391 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.393 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.444 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.499 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:25.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:25.552 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.606 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.660 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.714 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:25.768 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:25.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:25.826 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:25.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:25.883 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:34:25.884 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:25.910 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:26.007 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:26.034 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:26.101 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:26.197 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:26.225 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:26.252 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:26.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:26.376 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:26.378 DEBUG:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:26.431 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:34:26.431 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:34:26.431 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:34:26.433 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:34:26.434 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:34:26.434 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:34:26.435 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:34:26.436 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:34:26.436 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:34:26.473 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:34:26.474 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:34:26.474 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:34:26.474 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:34:26.476 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:34:26.476 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:34:26.476 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:34:26.478 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:34:26.478 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:34:26.478 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:34:26.580 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.582+0000 7f19d397c640 10 monclient: get_monmap_and_config 2024-03-29T11:34:26.580 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.582+0000 7f19d397c640 10 monclient: build_initial_monmap 2024-03-29T11:34:26.580 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.582+0000 7f19d397c640 10 monclient: monmap: 2024-03-29T11:34:26.580 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-29T11:34:26.580 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:26.581 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T11:34:26.584107+0000 2024-03-29T11:34:26.581 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T11:34:26.584107+0000 2024-03-29T11:34:26.581 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:26.581 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:26.581 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:26.581 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:26.581 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:26.581 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:26.581 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.583+0000 7f19d397c640 1 Processor -- start 2024-03-29T11:34:26.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.583+0000 7f19d397c640 1 -- start start 2024-03-29T11:34:26.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.583+0000 7f19d397c640 10 monclient: init 2024-03-29T11:34:26.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.583+0000 7f19d397c640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.583+0000 7f19d397c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.583+0000 7f19d397c640 10 monclient(hunting): picked mon.noname-b con 0x7f19cc09ac40 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.583+0000 7f19d397c640 10 monclient(hunting): picked mon.noname-a con 0x7f19cc09c870 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.584+0000 7f19d397c640 10 monclient(hunting): picked mon.noname-c con 0x7f19cc0b4360 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.584+0000 7f19d397c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc09dec0 con 0x7f19cc09c870 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.584+0000 7f19d397c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc09da30 con 0x7f19cc0b4360 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.584+0000 7f19d397c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc0b39c0 con 0x7f19cc09ac40 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.584+0000 7f19d397c640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.584+0000 7f19d397c640 10 monclient(hunting): authenticate will time out at 848.710913s 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.584+0000 7f19d0f17640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f19cc09c870 0x7f19cc13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:52764/0 (socket says 172.21.15.155:52764) 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.584+0000 7f19d1718640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f19cc09ac40 0x7f19cc0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:59366/0 (socket says 172.21.15.155:59366) 2024-03-29T11:34:26.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:26.584+0000 7f19d0f17640 1 -- 172.21.15.155:0/1602102498 learned_addr learned my addr 172.21.15.155:0/1602102498 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:29.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 10 monclient(hunting): tick 2024-03-29T11:34:29.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:34:29.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 1 monclient(hunting): continuing hunt 2024-03-29T11:34:29.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:34:29.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 1 -- 172.21.15.155:0/1602102498 >> v1:172.21.15.155:6790/0 conn(0x7f19cc09ac40 legacy=0x7f19cc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:29.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 1 -- 172.21.15.155:0/1602102498 >> v1:172.21.15.183:6789/0 conn(0x7f19cc0b4360 legacy=0x7f19cc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:29.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 1 -- 172.21.15.155:0/1602102498 >> v1:172.21.15.155:6789/0 conn(0x7f19cc09c870 legacy=0x7f19cc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:29.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:29.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f19b00036e0 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:29.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f19b00072d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:29.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f19b000aff0 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:29.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 1 -- 172.21.15.155:0/1602102498 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc0b5160 con 0x7f19b00036e0 2024-03-29T11:34:29.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 1 -- 172.21.15.155:0/1602102498 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b000fdf0 con 0x7f19b000aff0 2024-03-29T11:34:29.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 1 -- 172.21.15.155:0/1602102498 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b0011030 con 0x7f19b00072d0 2024-03-29T11:34:29.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:29.585+0000 7f19c2ffd640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.756+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935789670 0 0) 0x7f19cc0b5160 con 0x7f19b00036e0 2024-03-29T11:34:31.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.756+0000 7f19c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.756+0000 7f19c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.756+0000 7f19c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.756+0000 7f19c3fff640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.756+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b8003520 con 0x7f19b00036e0 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.756+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721163181 0 0) 0x7f19b000fdf0 con 0x7f19b000aff0 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19cc0b5160 con 0x7f19b000aff0 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 850245784 0 0) 0x7f19cc0b5160 con 0x7f19b000aff0 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 >> v1:172.21.15.155:6790/0 conn(0x7f19b00072d0 legacy=0x7f19b0009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 >> v1:172.21.15.155:6789/0 conn(0x7f19b00036e0 legacy=0x7f19b0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 1 monclient: found mon.noname-c 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b0012290 con 0x7f19b000aff0 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f19c4001a10 con 0x7f19b000aff0 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient: dump: 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19d397c640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19d397c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19d1f19640 1 -- 172.21.15.155:0/1602102498 >> v1:172.21.15.183:6789/0 conn(0x7f19b000aff0 legacy=0x7f19b000d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19d1f19640 1 -- 172.21.15.155:0/1602102498 >> v1:172.21.15.183:6789/0 conn(0x7f19b000aff0 legacy=0x7f19b000d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19d1f19640 1 --1- 172.21.15.155:0/1602102498 >> v1:172.21.15.183:6789/0 conn(0x7f19b000aff0 0x7f19b000d460 :-1 s=OPENED pgs=11 cs=1 l=1).handle_message read tag failed 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19d1f19640 1 --1- 172.21.15.155:0/1602102498 >> v1:172.21.15.183:6789/0 conn(0x7f19b000aff0 0x7f19b000d460 :-1 s=OPENED pgs=11 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient: ms_handle_reset current mon v1:172.21.15.183:6789/0 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 >> v1:172.21.15.183:6789/0 conn(0x7f19b000aff0 legacy=0x7f19b000d460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient(hunting): picked mon.c con 0x7f19b00072d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient(hunting): picked mon.a con 0x7f19b00036e0 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient(hunting): picked mon.b con 0x7f19b800b090 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f19cc0b5160 con 0x7f19b00036e0 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f19b000fdf0 con 0x7f19b800b090 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f19b8010ef0 con 0x7f19b00072d0 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.757+0000 7f19c3fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.183:6789/0 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484474697 0 0) 0x7f19b000fdf0 con 0x7f19b800b090 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 --> v1:172.21.15.183:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f19b8011080 con 0x7f19b800b090 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1536359553 0 0) 0x7f19b8011080 con 0x7f19b800b090 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 >> v1:172.21.15.155:6790/0 conn(0x7f19b00072d0 legacy=0x7f19b8003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 >> v1:172.21.15.155:6789/0 conn(0x7f19b00036e0 legacy=0x7f19b8009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 1 monclient: found mon.b 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f19b8012210 con 0x7f19b800b090 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f19c4002030 con 0x7f19b800b090 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 10 monclient: dump: 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19d397c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 1 -- 172.21.15.155:0/1602102498 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19c4004af0 con 0x7f19b800b090 2024-03-29T11:34:31.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19d397c640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:31.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.758+0000 7f19d397c640 4 set_mon_vals no callback set 2024-03-29T11:34:31.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.759+0000 7f19d397c640 10 monclient: shutdown 2024-03-29T11:34:31.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.759+0000 7f19d397c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.759+0000 7f19d397c640 1 -- 172.21.15.155:0/1602102498 >> v1:172.21.15.183:6789/0 conn(0x7f19b800b090 legacy=0x7f19b800d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.759+0000 7f19d397c640 1 -- 172.21.15.155:0/1602102498 shutdown_connections 2024-03-29T11:34:31.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.759+0000 7f19d397c640 1 -- 172.21.15.155:0/1602102498 >> 172.21.15.155:0/1602102498 conn(0x7f19cc066b00 msgr2=0x7f19cc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.759+0000 7f19d397c640 1 -- 172.21.15.155:0/1602102498 shutdown_connections 2024-03-29T11:34:31.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.759+0000 7f19d397c640 1 -- 172.21.15.155:0/1602102498 wait complete. 2024-03-29T11:34:31.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.760+0000 7f19d397c640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.760+0000 7f19d397c640 10 monclient: monmap: 2024-03-29T11:34:31.769 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-29T11:34:31.769 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:31.769 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-03-29T11:34:31.769 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-03-29T11:34:31.769 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.769 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.761+0000 7f19d397c640 1 librados: starting msgr at 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.761+0000 7f19d397c640 1 librados: starting objecter 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.761+0000 7f19d397c640 1 Processor -- start 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.761+0000 7f19d397c640 1 -- start start 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.761+0000 7f19d397c640 1 librados: setting wanted keys 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.761+0000 7f19d397c640 1 librados: calling monclient init 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.761+0000 7f19d397c640 10 monclient: init 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d397c640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d397c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d397c640 10 monclient(hunting): picked mon.noname-b con 0x7f19b00036e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d397c640 10 monclient(hunting): picked mon.noname-c con 0x7f19b00072d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d397c640 10 monclient(hunting): picked mon.noname-a con 0x7f19b800b090 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d397c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc13ffa0 con 0x7f19b800b090 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d1718640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f19b00036e0 0x7f19cc1459b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:33686/0 (socket says 172.21.15.155:33686) 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d1718640 1 -- 172.21.15.155:0/1251817595 learned_addr learned my addr 172.21.15.155:0/1251817595 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d397c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc142cc0 con 0x7f19b00036e0 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d397c640 1 -- 172.21.15.155:0/1251817595 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc1408d0 con 0x7f19b00072d0 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d397c640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d397c640 10 monclient(hunting): authenticate will time out at 853.889518s 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 10 client.?.objecter ms_handle_connect 0x7f19b00072d0 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 10 client.?.objecter ms_handle_connect 0x7f19b00036e0 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 10 client.?.objecter ms_handle_connect 0x7f19b800b090 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d0f17640 1 -- 172.21.15.155:0/1251817595 >> v1:172.21.15.155:6790/0 conn(0x7f19b00072d0 legacy=0x7f19cc1d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d0f17640 1 -- 172.21.15.155:0/1251817595 >> v1:172.21.15.155:6790/0 conn(0x7f19b00072d0 legacy=0x7f19cc1d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d0f17640 1 --1- 172.21.15.155:0/1251817595 >> v1:172.21.15.155:6790/0 conn(0x7f19b00072d0 0x7f19cc1d74c0 :-1 s=OPENED pgs=18 cs=1 l=1).handle_message read tag failed 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19d0f17640 1 --1- 172.21.15.155:0/1251817595 >> v1:172.21.15.155:6790/0 conn(0x7f19b00072d0 0x7f19cc1d74c0 :-1 s=OPENED pgs=18 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.155:6790/0 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035900668 0 0) 0x7f19cc13ffa0 con 0x7f19b800b090 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.762+0000 7f19c1ffb640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b0011ae0 con 0x7f19b800b090 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2884689620 0 0) 0x7f19b0011ae0 con 0x7f19b800b090 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19cc13ffa0 con 0x7f19b800b090 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f19c4002010 con 0x7f19b800b090 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient(hunting): dump: 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3942057969 0 0) 0x7f19cc13ffa0 con 0x7f19b800b090 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 >> v1:172.21.15.155:6790/0 conn(0x7f19b00072d0 legacy=0x7f19cc1d74c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 >> v1:172.21.15.183:6789/0 conn(0x7f19b00036e0 legacy=0x7f19cc1459b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 monclient: found mon.a 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc141020 con 0x7f19b800b090 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19d397c640 5 monclient: authenticate success, global_id 4110 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19d397c640 10 monclient: _renew_subs 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19d397c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19d397c640 1 -- 172.21.15.155:0/1251817595 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19cc1460c0 con 0x7f19b800b090 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19c4004c60 con 0x7f19b800b090 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f19c4004f70 con 0x7f19b800b090 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 10 monclient: dump: 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f19c4005520 con 0x7f19b800b090 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.763+0000 7f19c37fe640 4 set_mon_vals no callback set 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.764+0000 7f19d397c640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.764+0000 7f19d397c640 10 monclient: _renew_subs 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.764+0000 7f19d397c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.764+0000 7f19d397c640 1 -- 172.21.15.155:0/1251817595 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19cc146550 con 0x7f19b800b090 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.764+0000 7f19d397c640 1 librados: init done 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.764+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 4282386496 0 0) 0x7f19c4005d40 con 0x7f19b800b090 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.765+0000 7f19c1ffb640 10 client.4110.objecter ms_dispatch 0x7f19cc000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.765+0000 7f19c1ffb640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.765+0000 7f19c1ffb640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.765+0000 7f19c1ffb640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.765+0000 7f19d397c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.765+0000 7f19d397c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.765+0000 7f19d397c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.765+0000 7f19d397c640 1 -- 172.21.15.155:0/1251817595 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19cc1d7310 con 0x7f19b800b090 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.768+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f19c4005f80 con 0x7f19b800b090 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.768+0000 7f19c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.768+0000 7f19c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:31.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.844+0000 7f19d397c640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:31.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.844+0000 7f19d397c640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:31.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.844+0000 7f19d397c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:31.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.844+0000 7f19d397c640 1 -- 172.21.15.155:0/1251817595 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f19cc0b49d0 con 0x7f19b800b090 2024-03-29T11:34:31.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.846+0000 7f19c1ffb640 1 -- 172.21.15.155:0/1251817595 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f19c4006750 con 0x7f19b800b090 2024-03-29T11:34:31.844 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.846+0000 7f19c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:31.844 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.846+0000 7f19c1ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:34:31.844 INFO:teuthology.orchestra.run.smithi155.stderr:adjusted tunables profile to default 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 10 librados: watch_flush enter 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 10 librados: watch_flush exit 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 10 client.4110.objecter successfully canceled tick 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 10 monclient: shutdown 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 1 -- 172.21.15.155:0/1251817595 >> v1:172.21.15.155:6789/0 conn(0x7f19b800b090 legacy=0x7f19cc1e2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 1 -- 172.21.15.155:0/1251817595 shutdown_connections 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 1 -- 172.21.15.155:0/1251817595 >> 172.21.15.155:0/1251817595 conn(0x7f19cc066b00 msgr2=0x7f19cc13c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.846 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 1 -- 172.21.15.155:0/1251817595 shutdown_connections 2024-03-29T11:34:31.846 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 1 -- 172.21.15.155:0/1251817595 wait complete. 2024-03-29T11:34:31.846 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:31.847+0000 7f19d397c640 1 librados: shutdown 2024-03-29T11:34:31.856 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:34:31.856 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:34:31.856 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:34:31.856 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:34:31.883 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:34:31.883 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:34:31.950 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:34:31.950 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:34:32.016 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T11:34:32.016 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:34:32.083 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:34:32.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:34:32.111 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:34:32.111 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:34:32.178 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:34:32.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:34:32.237 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:32.240+0000 7f20ed843040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:32.245 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T11:34:32.245 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:34:32.268 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:32.271+0000 7fa948e10040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:32.293 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:32.296+0000 7f20ed843040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:34:32.312 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6 0 2024-03-29T11:34:32.322 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:32.325+0000 7fa948e10040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:34:32.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.454+0000 7fc273132640 10 monclient: get_monmap_and_config 2024-03-29T11:34:32.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.454+0000 7fc273132640 10 monclient: build_initial_monmap 2024-03-29T11:34:32.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.454+0000 7fc273132640 10 monclient: monmap: 2024-03-29T11:34:32.452 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:32.452 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.452 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:34:32.456010+0000 2024-03-29T11:34:32.453 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:34:32.456010+0000 2024-03-29T11:34:32.453 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:32.453 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.453 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:32.453 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:32.453 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:32.453 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.455+0000 7fc273132640 1 Processor -- start 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.455+0000 7fc273132640 1 -- start start 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.455+0000 7fc273132640 10 monclient: init 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.455+0000 7fc273132640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.455+0000 7fc273132640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.455+0000 7fc273132640 10 monclient(hunting): picked mon.noname-b con 0x7fc26c09ac40 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 10 monclient(hunting): picked mon.noname-c con 0x7fc26c09c870 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 10 monclient(hunting): picked mon.noname-a con 0x7fc26c0b4360 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26c09dec0 con 0x7fc26c0b4360 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26c09da30 con 0x7fc26c09c870 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26c0b39c0 con 0x7fc26c09ac40 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 10 monclient(hunting): authenticate will time out at 864.728254s 2024-03-29T11:34:32.463 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26bfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fc26c09c870 0x7fc26c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43576/0 (socket says 172.21.15.183:43576) 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26bfff640 1 -- 172.21.15.183:0/2904450973 learned_addr learned my addr 172.21.15.183:0/2904450973 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481260824 0 0) 0x7fc26c09da30 con 0x7fc26c09c870 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc25c003520 con 0x7fc26c09c870 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413139593 0 0) 0x7fc26c0b39c0 con 0x7fc26c09ac40 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26c09da30 con 0x7fc26c09ac40 2024-03-29T11:34:32.464 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901437516 0 0) 0x7fc264001490 con 0x7fc26c0b4360 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26c0b39c0 con 0x7fc26c0b4360 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1334854498 0 0) 0x7fc25c003520 con 0x7fc26c09c870 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 >> v1:172.21.15.155:6790/0 conn(0x7fc26c09ac40 legacy=0x7fc26c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 >> v1:172.21.15.155:6789/0 conn(0x7fc26c0b4360 legacy=0x7fc26c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c0b5160 con 0x7fc26c09c870 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:32.465 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc258001690 con 0x7fc26c09c870 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient: dump: 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.466 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc258001870 con 0x7fc26c09c870 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 1 -- 172.21.15.183:0/2904450973 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc258001c80 con 0x7fc26c09c870 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc273132640 4 set_mon_vals no callback set 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.456+0000 7fc26b7fe640 10 monclient: dump: 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.467 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.457+0000 7fc273132640 10 monclient: shutdown 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.457+0000 7fc273132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.457+0000 7fc273132640 1 -- 172.21.15.183:0/2904450973 >> v1:172.21.15.183:6789/0 conn(0x7fc26c09c870 legacy=0x7fc26c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.457+0000 7fc273132640 1 -- 172.21.15.183:0/2904450973 shutdown_connections 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.457+0000 7fc273132640 1 -- 172.21.15.183:0/2904450973 >> 172.21.15.183:0/2904450973 conn(0x7fc26c066b00 msgr2=0x7fc26c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:32.468 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.457+0000 7fc273132640 1 -- 172.21.15.183:0/2904450973 shutdown_connections 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.457+0000 7fc273132640 1 -- 172.21.15.183:0/2904450973 wait complete. 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.458+0000 7fc273132640 10 monclient: build_initial_monmap 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.458+0000 7fc273132640 10 monclient: monmap: 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.469 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.458+0000 7fc273132640 1 librados: starting msgr at 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.458+0000 7fc273132640 1 librados: starting objecter 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.458+0000 7fc273132640 1 Processor -- start 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.458+0000 7fc273132640 1 -- start start 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.458+0000 7fc273132640 1 librados: setting wanted keys 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.458+0000 7fc273132640 1 librados: calling monclient init 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.458+0000 7fc273132640 10 monclient: init 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 monclient(hunting): picked mon.noname-a con 0x7fc26c09ac40 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 monclient(hunting): picked mon.noname-c con 0x7fc26c0acf60 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 monclient(hunting): picked mon.noname-b con 0x7fc26c0b0800 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26c1da7b0 con 0x7fc26c09ac40 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26c0b0c70 con 0x7fc26c0b0800 2024-03-29T11:34:32.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26c1db350 con 0x7fc26c0acf60 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 monclient(hunting): authenticate will time out at 864.731286s 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2716cf640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fc26c0b0800 0x7fc26c1e1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43586/0 (socket says 172.21.15.183:43586) 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2716cf640 1 -- 172.21.15.183:0/2346745086 learned_addr learned my addr 172.21.15.183:0/2346745086 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 client.?.objecter ms_handle_connect 0x7fc26c0b0800 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 client.?.objecter ms_handle_connect 0x7fc26c09ac40 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 client.?.objecter ms_handle_connect 0x7fc26c0acf60 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865424713 0 0) 0x7fc26c0b0c70 con 0x7fc26c0b0800 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.471 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc250004eb0 con 0x7fc26c0b0800 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463357845 0 0) 0x7fc26c1db350 con 0x7fc26c0acf60 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26c0b0c70 con 0x7fc26c0acf60 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642014485 0 0) 0x7fc26c1da7b0 con 0x7fc26c09ac40 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26c1db350 con 0x7fc26c09ac40 2024-03-29T11:34:32.472 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3375887670 0 0) 0x7fc250004eb0 con 0x7fc26c0b0800 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc26c1da7b0 con 0x7fc26c0b0800 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc264001bc0 con 0x7fc26c0b0800 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): dump: 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4083028154 0 0) 0x7fc26c0b0c70 con 0x7fc26c0acf60 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc250004eb0 con 0x7fc26c0acf60 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc258001b80 con 0x7fc26c0acf60 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): dump: 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.474 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1182112610 0 0) 0x7fc26c1db350 con 0x7fc26c09ac40 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc26c0b0c70 con 0x7fc26c09ac40 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc260001870 con 0x7fc26c09ac40 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient(hunting): dump: 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.475 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 220374518 0 0) 0x7fc26c1da7b0 con 0x7fc26c0b0800 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 >> v1:172.21.15.155:6790/0 conn(0x7fc26c0acf60 legacy=0x7fc26c1dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 >> v1:172.21.15.155:6789/0 conn(0x7fc26c09ac40 legacy=0x7fc26c0af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient: _finish_hunting 0 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 monclient: found mon.b 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c1db4e0 con 0x7fc26c0b0800 2024-03-29T11:34:32.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient: _finish_auth 0 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 10 monclient: _check_auth_tickets 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc2697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 monclient: _renew_subs 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 1 -- 172.21.15.183:0/2346745086 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26c1e3240 con 0x7fc26c0b0800 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 monclient: _renew_subs 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 1 -- 172.21.15.183:0/2346745086 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26c1e35b0 con 0x7fc26c0b0800 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.459+0000 7fc273132640 1 librados: init done 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc264001fd0 con 0x7fc26c0b0800 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc26affd640 4 set_mon_vals no callback set 2024-03-29T11:34:32.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc264003990 con 0x7fc26c0b0800 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 10 monclient: dump: 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.478 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc264003f40 con 0x7fc26c0b0800 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2665762774 0 0) 0x7fc264006490 con 0x7fc26c0b0800 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 10 client.4111.objecter ms_dispatch 0x7fc26c000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc2697fa640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc273132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc273132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc273132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.460+0000 7fc273132640 1 -- 172.21.15.183:0/2346745086 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc24c005080 con 0x7fc26c0b0800 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.462+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc264005b40 con 0x7fc26c0b0800 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.462+0000 7fc2697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.462+0000 7fc2697fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:32.532 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.534+0000 7fc273132640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6", "id": 0}] 2024-03-29T11:34:32.532 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.534+0000 7fc273132640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6", "id": 0}] 2024-03-29T11:34:32.532 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.534+0000 7fc273132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.532 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.534+0000 7fc273132640 1 -- 172.21.15.183:0/2346745086 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6", "id": 0} v 0) v1 -- 0x7fc24c0058b0 con 0x7fc26c0b0800 2024-03-29T11:34:32.537 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.540+0000 7fc2697fa640 1 -- 172.21.15.183:0/2346745086 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3380914309 0 732707515) 0x7fc264004420 con 0x7fc26c0b0800 2024-03-29T11:34:32.538 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.540+0000 7fc2697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6", "id": 0}] 2024-03-29T11:34:32.538 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.540+0000 7fc2697fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:32.538 INFO:teuthology.orchestra.run.smithi183.stdout:0 2024-03-29T11:34:32.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 10 librados: watch_flush enter 2024-03-29T11:34:32.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 10 librados: watch_flush exit 2024-03-29T11:34:32.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T11:34:32.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 10 client.4111.objecter successfully canceled tick 2024-03-29T11:34:32.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 10 monclient: shutdown 2024-03-29T11:34:32.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:32.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 1 -- 172.21.15.183:0/2346745086 >> v1:172.21.15.183:6789/0 conn(0x7fc26c0b0800 legacy=0x7fc26c1e1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 1 -- 172.21.15.183:0/2346745086 shutdown_connections 2024-03-29T11:34:32.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 1 -- 172.21.15.183:0/2346745086 >> 172.21.15.183:0/2346745086 conn(0x7fc26c066b00 msgr2=0x7fc26c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:32.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 1 -- 172.21.15.183:0/2346745086 shutdown_connections 2024-03-29T11:34:32.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 1 -- 172.21.15.183:0/2346745086 wait complete. 2024-03-29T11:34:32.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.541+0000 7fc273132640 1 librados: shutdown 2024-03-29T11:34:32.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new acce00ab-e768-4e25-a833-702e26eeda25 1 2024-03-29T11:34:32.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.650+0000 7fe49ec4a640 10 monclient: get_monmap_and_config 2024-03-29T11:34:32.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.650+0000 7fe49ec4a640 10 monclient: build_initial_monmap 2024-03-29T11:34:32.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.650+0000 7fe49ec4a640 10 monclient: monmap: 2024-03-29T11:34:32.649 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:32.649 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.649 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:34:32.652239+0000 2024-03-29T11:34:32.649 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:34:32.652239+0000 2024-03-29T11:34:32.649 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:32.649 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.649 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:32.649 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:32.649 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:32.649 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.651+0000 7fe49ec4a640 1 Processor -- start 2024-03-29T11:34:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.651+0000 7fe49ec4a640 1 -- start start 2024-03-29T11:34:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.651+0000 7fe49ec4a640 10 monclient: init 2024-03-29T11:34:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49ec4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49ec4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49ec4a640 10 monclient(hunting): picked mon.noname-c con 0x7fe49809ac40 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49ec4a640 10 monclient(hunting): picked mon.noname-b con 0x7fe49809c870 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49ec4a640 10 monclient(hunting): picked mon.noname-a con 0x7fe4980b4360 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49ec4a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49809dec0 con 0x7fe4980b4360 2024-03-29T11:34:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49ec4a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49809da30 con 0x7fe49809ac40 2024-03-29T11:34:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49d1e7640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fe4980b4360 0x7fe49814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.183:38158/0 (socket says 172.21.15.183:38158) 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49d1e7640 1 -- 172.21.15.183:0/3066791847 learned_addr learned my addr 172.21.15.183:0/3066791847 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe497fff640 1 --1- 172.21.15.183:0/3066791847 >> v1:172.21.15.155:6790/0 conn(0x7fe49809c870 0x7fe49813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.183:53866/0 (socket says 172.21.15.183:53866) 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49ec4a640 1 -- 172.21.15.183:0/3066791847 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4980b39c0 con 0x7fe49809c870 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49ec4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe49ec4a640 10 monclient(hunting): authenticate will time out at 864.924713s 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778807155 0 0) 0x7fe49809da30 con 0x7fe49809ac40 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe484003520 con 0x7fe49809ac40 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971167780 0 0) 0x7fe4980b39c0 con 0x7fe49809c870 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49809da30 con 0x7fe49809c870 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616533106 0 0) 0x7fe4900013c0 con 0x7fe4980b4360 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4980b39c0 con 0x7fe4980b4360 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4200769427 0 0) 0x7fe484003520 con 0x7fe49809ac40 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 >> v1:172.21.15.155:6790/0 conn(0x7fe49809c870 legacy=0x7fe49813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 >> v1:172.21.15.155:6789/0 conn(0x7fe4980b4360 legacy=0x7fe49814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:32.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 1 monclient: found mon.noname-c 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.652+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4980b5160 con 0x7fe49809ac40 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe49ec4a640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe488001690 con 0x7fe49809ac40 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 10 monclient: dump: 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.661 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe49ec4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe488001870 con 0x7fe49809ac40 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 1 -- 172.21.15.183:0/3066791847 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe488001c80 con 0x7fe49809ac40 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe4977fe640 10 monclient: dump: 2024-03-29T11:34:32.662 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe49ec4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe49ec4a640 4 set_mon_vals no callback set 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe49ec4a640 10 monclient: shutdown 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe49ec4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe49ec4a640 1 -- 172.21.15.183:0/3066791847 >> v1:172.21.15.183:6789/0 conn(0x7fe49809ac40 legacy=0x7fe4980b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe49ec4a640 1 -- 172.21.15.183:0/3066791847 shutdown_connections 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe49ec4a640 1 -- 172.21.15.183:0/3066791847 >> 172.21.15.183:0/3066791847 conn(0x7fe498066b00 msgr2=0x7fe498066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe49ec4a640 1 -- 172.21.15.183:0/3066791847 shutdown_connections 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.653+0000 7fe49ec4a640 1 -- 172.21.15.183:0/3066791847 wait complete. 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.654+0000 7fe49ec4a640 10 monclient: build_initial_monmap 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.654+0000 7fe49ec4a640 10 monclient: monmap: 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:32.664 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.655+0000 7fe49ec4a640 1 librados: starting msgr at 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.655+0000 7fe49ec4a640 1 librados: starting objecter 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.655+0000 7fe49ec4a640 1 Processor -- start 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.655+0000 7fe49ec4a640 1 -- start start 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.655+0000 7fe49ec4a640 1 librados: setting wanted keys 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.655+0000 7fe49ec4a640 1 librados: calling monclient init 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.655+0000 7fe49ec4a640 10 monclient: init 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49ec4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49ec4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49ec4a640 10 monclient(hunting): picked mon.noname-b con 0x7fe49809ac40 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49ec4a640 10 monclient(hunting): picked mon.noname-c con 0x7fe4981d7f60 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49ec4a640 10 monclient(hunting): picked mon.noname-a con 0x7fe4981398c0 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49ec4a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4980b18b0 con 0x7fe4981398c0 2024-03-29T11:34:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49ec4a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4981d73c0 con 0x7fe49809ac40 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49ec4a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4980b1ae0 con 0x7fe4981d7f60 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49ec4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49ec4a640 10 monclient(hunting): authenticate will time out at 864.928458s 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49c9e6640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe49809ac40 0x7fe4981d7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43608/0 (socket says 172.21.15.183:43608) 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe49c9e6640 1 -- 172.21.15.183:0/2317267005 learned_addr learned my addr 172.21.15.183:0/2317267005 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 client.?.objecter ms_handle_connect 0x7fe49809ac40 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 client.?.objecter ms_handle_connect 0x7fe4981398c0 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 client.?.objecter ms_handle_connect 0x7fe4981d7f60 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984008941 0 0) 0x7fe4981d73c0 con 0x7fe49809ac40 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe478004eb0 con 0x7fe49809ac40 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656851420 0 0) 0x7fe4980b18b0 con 0x7fe4981398c0 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4981d73c0 con 0x7fe4981398c0 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926915010 0 0) 0x7fe4980b1ae0 con 0x7fe4981d7f60 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.667 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4980b18b0 con 0x7fe4981d7f60 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2408827102 0 0) 0x7fe478004eb0 con 0x7fe49809ac40 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4980b1ae0 con 0x7fe49809ac40 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe488001c80 con 0x7fe49809ac40 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): dump: 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.668 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3256124993 0 0) 0x7fe4980b18b0 con 0x7fe4981d7f60 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe478004eb0 con 0x7fe4981d7f60 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3778684743 0 0) 0x7fe4981d73c0 con 0x7fe4981398c0 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4980b18b0 con 0x7fe4981398c0 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe480001930 con 0x7fe4981d7f60 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.656+0000 7fe4957fa640 10 monclient(hunting): dump: 2024-03-29T11:34:32.669 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe490001770 con 0x7fe4981398c0 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 monclient(hunting): dump: 2024-03-29T11:34:32.670 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 166323758 0 0) 0x7fe4980b1ae0 con 0x7fe49809ac40 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 >> v1:172.21.15.155:6790/0 conn(0x7fe4981d7f60 legacy=0x7fe4981391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 >> v1:172.21.15.155:6789/0 conn(0x7fe4981398c0 legacy=0x7fe4980b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 monclient: _finish_hunting 0 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 1 monclient: found mon.b 2024-03-29T11:34:32.671 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4980acd30 con 0x7fe49809ac40 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 monclient: _finish_auth 0 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 monclient: _check_auth_tickets 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe49ec4a640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe49ec4a640 10 monclient: _renew_subs 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe49ec4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe49ec4a640 1 -- 172.21.15.183:0/2317267005 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4980b0830 con 0x7fe49809ac40 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe49ec4a640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe49ec4a640 10 monclient: _renew_subs 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe49ec4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe49ec4a640 1 -- 172.21.15.183:0/2317267005 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4980b0cc0 con 0x7fe49809ac40 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe49ec4a640 1 librados: init done 2024-03-29T11:34:32.672 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe488003960 con 0x7fe49809ac40 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe488003cf0 con 0x7fe49809ac40 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 monclient: dump: 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.673 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe496ffd640 4 set_mon_vals no callback set 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe488004260 con 0x7fe49809ac40 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3978133984 0 0) 0x7fe488006850 con 0x7fe49809ac40 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 10 client.4117.objecter ms_dispatch 0x7fe498000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.657+0000 7fe4957fa640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.658+0000 7fe49ec4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.658+0000 7fe49ec4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.658+0000 7fe49ec4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.658+0000 7fe49ec4a640 1 -- 172.21.15.183:0/2317267005 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47c005080 con 0x7fe49809ac40 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.660+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe488006b30 con 0x7fe49809ac40 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.660+0000 7fe4957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.660+0000 7fe4957fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:32.730 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.732+0000 7fe49ec4a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "acce00ab-e768-4e25-a833-702e26eeda25", "id": 1}] 2024-03-29T11:34:32.730 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.732+0000 7fe49ec4a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "acce00ab-e768-4e25-a833-702e26eeda25", "id": 1}] 2024-03-29T11:34:32.730 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.732+0000 7fe49ec4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:32.730 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.732+0000 7fe49ec4a640 1 -- 172.21.15.183:0/2317267005 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "acce00ab-e768-4e25-a833-702e26eeda25", "id": 1} v 0) v1 -- 0x7fe47c0058b0 con 0x7fe49809ac40 2024-03-29T11:34:32.737 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.740+0000 7fe4957fa640 1 -- 172.21.15.183:0/2317267005 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "acce00ab-e768-4e25-a833-702e26eeda25", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2798145624 0 940483276) 0x7fe488005910 con 0x7fe49809ac40 2024-03-29T11:34:32.737 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.740+0000 7fe4957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "acce00ab-e768-4e25-a833-702e26eeda25", "id": 1}] 2024-03-29T11:34:32.737 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.740+0000 7fe4957fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:32.738 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-29T11:34:32.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.740+0000 7fe49ec4a640 10 librados: watch_flush enter 2024-03-29T11:34:32.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.741+0000 7fe49ec4a640 10 librados: watch_flush exit 2024-03-29T11:34:32.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.741+0000 7fe49ec4a640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:34:32.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.741+0000 7fe49ec4a640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:34:32.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.741+0000 7fe49ec4a640 10 monclient: shutdown 2024-03-29T11:34:32.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.741+0000 7fe49ec4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:32.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.741+0000 7fe49ec4a640 1 -- 172.21.15.183:0/2317267005 >> v1:172.21.15.183:6789/0 conn(0x7fe49809ac40 legacy=0x7fe4981d7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.740 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.741+0000 7fe49ec4a640 1 -- 172.21.15.183:0/2317267005 shutdown_connections 2024-03-29T11:34:32.740 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.741+0000 7fe49ec4a640 1 -- 172.21.15.183:0/2317267005 >> 172.21.15.183:0/2317267005 conn(0x7fe498066b00 msgr2=0x7fe4981420b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:32.740 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.741+0000 7fe49ec4a640 1 -- 172.21.15.183:0/2317267005 shutdown_connections 2024-03-29T11:34:32.740 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.741+0000 7fe49ec4a640 1 -- 172.21.15.183:0/2317267005 wait complete. 2024-03-29T11:34:32.740 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.741+0000 7fe49ec4a640 1 librados: shutdown 2024-03-29T11:34:32.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new bca8f954-a16c-417f-a7c0-d4109884c11b 2 2024-03-29T11:34:32.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.850+0000 7efec237b640 10 monclient: get_monmap_and_config 2024-03-29T11:34:32.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.850+0000 7efec237b640 10 monclient: build_initial_monmap 2024-03-29T11:34:32.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.850+0000 7efec237b640 10 monclient: monmap: 2024-03-29T11:34:32.848 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:32.848 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.848 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:34:32.851950+0000 2024-03-29T11:34:32.848 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:34:32.851950+0000 2024-03-29T11:34:32.848 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:32.849 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.849 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:32.849 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:32.849 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:32.849 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.858 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 1 Processor -- start 2024-03-29T11:34:32.858 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 1 -- start start 2024-03-29T11:34:32.858 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 10 monclient: init 2024-03-29T11:34:32.858 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:32.858 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:32.858 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 10 monclient(hunting): picked mon.noname-a con 0x7efebc09ac40 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:32.858 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 10 monclient(hunting): picked mon.noname-b con 0x7efebc09c870 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:32.858 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 10 monclient(hunting): picked mon.noname-c con 0x7efebc0b4360 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:32.858 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efebc09dec0 con 0x7efebc09ac40 2024-03-29T11:34:32.858 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efebc09da30 con 0x7efebc0b4360 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efebc0b39c0 con 0x7efebc09c870 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.851+0000 7efec237b640 10 monclient(hunting): authenticate will time out at 865.124106s 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efec0918640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7efebc0b4360 0x7efebc14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43614/0 (socket says 172.21.15.183:43614) 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efec0918640 1 -- 172.21.15.183:0/3261413349 learned_addr learned my addr 172.21.15.183:0/3261413349 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690167335 0 0) 0x7efebc09da30 con 0x7efebc0b4360 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea4003520 con 0x7efebc0b4360 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515590381 0 0) 0x7efeac001110 con 0x7efebc09ac40 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efebc09da30 con 0x7efebc09ac40 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673543574 0 0) 0x7efebc0b39c0 con 0x7efebc09c870 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeac001110 con 0x7efebc09c870 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348789430 0 0) 0x7efebc09da30 con 0x7efebc09ac40 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 >> v1:172.21.15.155:6790/0 conn(0x7efebc09c870 legacy=0x7efebc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 >> v1:172.21.15.183:6789/0 conn(0x7efebc0b4360 legacy=0x7efebc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:32.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 monclient: found mon.noname-a 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efebc0b5160 con 0x7efebc09ac40 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efec237b640 5 monclient: authenticate success, global_id 4125 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7efeac001660 con 0x7efebc09ac40 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient: dump: 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.861 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efec237b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeac001840 con 0x7efebc09ac40 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 1 -- 172.21.15.183:0/3261413349 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7efeac001c90 con 0x7efebc09ac40 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efec237b640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:32.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efec237b640 4 set_mon_vals no callback set 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.852+0000 7efebaffd640 10 monclient: dump: 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.853+0000 7efec237b640 10 monclient: shutdown 2024-03-29T11:34:32.863 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.853+0000 7efec237b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.853+0000 7efec237b640 1 -- 172.21.15.183:0/3261413349 >> v1:172.21.15.155:6789/0 conn(0x7efebc09ac40 legacy=0x7efebc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.853+0000 7efec237b640 1 -- 172.21.15.183:0/3261413349 shutdown_connections 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.853+0000 7efec237b640 1 -- 172.21.15.183:0/3261413349 >> 172.21.15.183:0/3261413349 conn(0x7efebc066b00 msgr2=0x7efebc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.853+0000 7efec237b640 1 -- 172.21.15.183:0/3261413349 shutdown_connections 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.853+0000 7efec237b640 1 -- 172.21.15.183:0/3261413349 wait complete. 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.854+0000 7efec237b640 10 monclient: build_initial_monmap 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.854+0000 7efec237b640 10 monclient: monmap: 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.864 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.854+0000 7efec237b640 1 librados: starting msgr at 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.854+0000 7efec237b640 1 librados: starting objecter 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.854+0000 7efec237b640 1 Processor -- start 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.854+0000 7efec237b640 1 -- start start 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.854+0000 7efec237b640 1 librados: setting wanted keys 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.854+0000 7efec237b640 1 librados: calling monclient init 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.854+0000 7efec237b640 10 monclient: init 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 10 monclient(hunting): picked mon.noname-a con 0x7efebc09ac40 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 10 monclient(hunting): picked mon.noname-b con 0x7efebc1462f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:32.865 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 10 monclient(hunting): picked mon.noname-c con 0x7efebc0b1830 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efebc145c80 con 0x7efebc09ac40 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efebc144c40 con 0x7efebc1462f0 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efebc145ed0 con 0x7efebc0b1830 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 10 monclient(hunting): authenticate will time out at 865.127322s 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efebb7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7efebc1462f0 0x7efebc1da7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43624/0 (socket says 172.21.15.183:43624) 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efebb7fe640 1 -- 172.21.15.183:0/3873744487 learned_addr learned my addr 172.21.15.183:0/3873744487 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 client.?.objecter ms_handle_connect 0x7efebc1462f0 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 client.?.objecter ms_handle_connect 0x7efebc09ac40 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 client.?.objecter ms_handle_connect 0x7efebc0b1830 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962566043 0 0) 0x7efebc145c80 con 0x7efebc09ac40 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea8004eb0 con 0x7efebc09ac40 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683546057 0 0) 0x7efebc144c40 con 0x7efebc1462f0 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efebc145c80 con 0x7efebc1462f0 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801075350 0 0) 0x7efebc145ed0 con 0x7efebc0b1830 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efebc144c40 con 0x7efebc0b1830 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2092151794 0 0) 0x7efea8004eb0 con 0x7efebc09ac40 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efebc145ed0 con 0x7efebc09ac40 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7efeac001c10 con 0x7efebc09ac40 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.868 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1034674719 0 0) 0x7efebc145c80 con 0x7efebc1462f0 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efea8004eb0 con 0x7efebc1462f0 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 768791941 0 0) 0x7efebc144c40 con 0x7efebc0b1830 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efebc145c80 con 0x7efebc0b1830 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7efeb0001a10 con 0x7efebc1462f0 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:32.869 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7efeb40019f0 con 0x7efebc0b1830 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 991839553 0 0) 0x7efebc145ed0 con 0x7efebc09ac40 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 >> v1:172.21.15.155:6790/0 conn(0x7efebc0b1830 legacy=0x7efebc1daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 >> v1:172.21.15.183:6789/0 conn(0x7efebc1462f0 legacy=0x7efebc1da7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 monclient: found mon.a 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efebc1445f0 con 0x7efebc09ac40 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efeb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 5 monclient: authenticate success, global_id 4128 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 10 monclient: _renew_subs 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.855+0000 7efec237b640 1 -- 172.21.15.183:0/3873744487 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efebc144840 con 0x7efebc09ac40 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efec237b640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efec237b640 10 monclient: _renew_subs 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efec237b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efec237b640 1 -- 172.21.15.183:0/3873744487 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efebc143c50 con 0x7efebc09ac40 2024-03-29T11:34:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efec237b640 1 librados: init done 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeac003960 con 0x7efebc09ac40 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7efeac003c70 con 0x7efebc09ac40 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeba7fc640 4 set_mon_vals no callback set 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 10 monclient: dump: 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:32.873 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efeac004160 con 0x7efebc09ac40 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 4287854911 0 0) 0x7efeac004830 con 0x7efebc09ac40 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 10 client.4128.objecter ms_dispatch 0x7efebc000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 3 client.4128.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 3 client.4128.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efeb8ff9640 20 client.4128.objecter dump_active .. 0 homeless 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efec237b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efec237b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efec237b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.856+0000 7efec237b640 1 -- 172.21.15.183:0/3873744487 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c005080 con 0x7efebc09ac40 2024-03-29T11:34:32.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.859+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efeac004410 con 0x7efebc09ac40 2024-03-29T11:34:32.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.859+0000 7efeb8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.859+0000 7efeb8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:32.880 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:32.884+0000 7fa948e10040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:34:32.882 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:32.885+0000 7f20ed843040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:34:32.928 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.931+0000 7efec237b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bca8f954-a16c-417f-a7c0-d4109884c11b", "id": 2}] 2024-03-29T11:34:32.928 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.931+0000 7efec237b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bca8f954-a16c-417f-a7c0-d4109884c11b", "id": 2}] 2024-03-29T11:34:32.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.931+0000 7efec237b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:32.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.931+0000 7efec237b640 1 -- 172.21.15.183:0/3873744487 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bca8f954-a16c-417f-a7c0-d4109884c11b", "id": 2} v 0) v1 -- 0x7efe9c0058b0 con 0x7efebc09ac40 2024-03-29T11:34:32.930 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.932+0000 7efeb8ff9640 1 -- 172.21.15.183:0/3873744487 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bca8f954-a16c-417f-a7c0-d4109884c11b", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1760704172 0 216599125) 0x7efeac004a60 con 0x7efebc09ac40 2024-03-29T11:34:32.930 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.932+0000 7efeb8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bca8f954-a16c-417f-a7c0-d4109884c11b", "id": 2}] 2024-03-29T11:34:32.930 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.932+0000 7efeb8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:32.930 INFO:teuthology.orchestra.run.smithi183.stdout:2 2024-03-29T11:34:32.931 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.933+0000 7efec237b640 10 librados: watch_flush enter 2024-03-29T11:34:32.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.933+0000 7efec237b640 10 librados: watch_flush exit 2024-03-29T11:34:32.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.933+0000 7efec237b640 20 client.4128.objecter shutdown clearing up homeless session... 2024-03-29T11:34:32.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.933+0000 7efec237b640 10 client.4128.objecter successfully canceled tick 2024-03-29T11:34:32.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.933+0000 7efec237b640 10 monclient: shutdown 2024-03-29T11:34:32.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.933+0000 7efec237b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:32.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.933+0000 7efec237b640 1 -- 172.21.15.183:0/3873744487 >> v1:172.21.15.155:6789/0 conn(0x7efebc09ac40 legacy=0x7efebc0b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.933+0000 7efec237b640 1 -- 172.21.15.183:0/3873744487 shutdown_connections 2024-03-29T11:34:32.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.933+0000 7efec237b640 1 -- 172.21.15.183:0/3873744487 >> 172.21.15.183:0/3873744487 conn(0x7efebc066b00 msgr2=0x7efebc13af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:32.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.933+0000 7efec237b640 1 -- 172.21.15.183:0/3873744487 shutdown_connections 2024-03-29T11:34:32.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.934+0000 7efec237b640 1 -- 172.21.15.183:0/3873744487 wait complete. 2024-03-29T11:34:32.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:32.934+0000 7efec237b640 1 librados: shutdown 2024-03-29T11:34:32.935 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:32.939+0000 7fa948e10040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:34:32.939 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:32.942+0000 7f20ed843040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:34:32.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new a27cee2b-5d5c-421f-bcc5-16b9e3f5b559 3 2024-03-29T11:34:32.981 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:32.985+0000 7fa948e10040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:34:32.987 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:32.990+0000 7f20ed843040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:34:33.034 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:33.038+0000 7fa948e10040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:34:33.041 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.043+0000 7f86c5f4d640 10 monclient: get_monmap_and_config 2024-03-29T11:34:33.041 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.043+0000 7f86c5f4d640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.041 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.044+0000 7f86c5f4d640 10 monclient: monmap: 2024-03-29T11:34:33.041 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:33.041 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.041 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:34:33.045363+0000 2024-03-29T11:34:33.041 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:34:33.045363+0000 2024-03-29T11:34:33.042 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.042 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.042 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:33.042 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:33.042 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:33.042 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.046 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.044+0000 7f86c5f4d640 1 Processor -- start 2024-03-29T11:34:33.046 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.044+0000 7f86c5f4d640 1 -- start start 2024-03-29T11:34:33.046 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.044+0000 7f86c5f4d640 10 monclient: init 2024-03-29T11:34:33.046 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86c5f4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.046 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86c5f4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:33.046 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86c5f4d640 10 monclient(hunting): picked mon.noname-a con 0x7f86c00af360 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:33.046 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86c5f4d640 10 monclient(hunting): picked mon.noname-b con 0x7f86c006af60 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:33.046 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86c5f4d640 10 monclient(hunting): picked mon.noname-c con 0x7f86c0145e60 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:33.046 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86c5f4d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c00b0520 con 0x7f86c00af360 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86c5f4d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c00aff20 con 0x7f86c0145e60 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86c5f4d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c00b1da0 con 0x7f86c006af60 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86c5f4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86c5f4d640 10 monclient(hunting): authenticate will time out at 865.317669s 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86bffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f86c0145e60 0x7f86c014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43636/0 (socket says 172.21.15.183:43636) 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86bffff640 1 -- 172.21.15.183:0/4239602713 learned_addr learned my addr 172.21.15.183:0/4239602713 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345445254 0 0) 0x7f86c00aff20 con 0x7f86c0145e60 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86ac003520 con 0x7f86c0145e60 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487539085 0 0) 0x7f86c00b0520 con 0x7f86c00af360 2024-03-29T11:34:33.047 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c00aff20 con 0x7f86c00af360 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715331071 0 0) 0x7f86c00b1da0 con 0x7f86c006af60 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.045+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c00b0520 con 0x7f86c006af60 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 550421615 0 0) 0x7f86c00aff20 con 0x7f86c00af360 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 >> v1:172.21.15.155:6790/0 conn(0x7f86c006af60 legacy=0x7f86c0145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 >> v1:172.21.15.183:6789/0 conn(0x7f86c0145e60 legacy=0x7f86c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 1 monclient: found mon.noname-a 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c00b2080 con 0x7f86c00af360 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f86b0001670 con 0x7f86c00af360 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 5 monclient: authenticate success, global_id 4131 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 10 monclient: dump: 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.049 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86b0001850 con 0x7f86c00af360 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:33.050 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 4 set_mon_vals no callback set 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 1 -- 172.21.15.183:0/4239602713 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f86b0001c20 con 0x7f86c00af360 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86beffd640 10 monclient: dump: 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 10 monclient: shutdown 2024-03-29T11:34:33.052 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.052 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 1 -- 172.21.15.183:0/4239602713 >> v1:172.21.15.155:6789/0 conn(0x7f86c00af360 legacy=0x7f86c006c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.052 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 1 -- 172.21.15.183:0/4239602713 shutdown_connections 2024-03-29T11:34:33.052 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 1 -- 172.21.15.183:0/4239602713 >> 172.21.15.183:0/4239602713 conn(0x7f86c00b58f0 msgr2=0x7f86c00b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.052 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 1 -- 172.21.15.183:0/4239602713 shutdown_connections 2024-03-29T11:34:33.052 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.046+0000 7f86c5f4d640 1 -- 172.21.15.183:0/4239602713 wait complete. 2024-03-29T11:34:33.052 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:33.047+0000 7f20ed843040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.047+0000 7f86c5f4d640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.047+0000 7f86c5f4d640 10 monclient: monmap: 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.058 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.047+0000 7f86c5f4d640 1 librados: starting msgr at 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.047+0000 7f86c5f4d640 1 librados: starting objecter 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.047+0000 7f86c5f4d640 1 Processor -- start 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 1 -- start start 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 1 librados: setting wanted keys 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 1 librados: calling monclient init 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 10 monclient: init 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 10 monclient(hunting): picked mon.noname-b con 0x7f86c006af60 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 10 monclient(hunting): picked mon.noname-c con 0x7f86c00af360 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 10 monclient(hunting): picked mon.noname-a con 0x7f86c0145e60 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c013e0c0 con 0x7f86c0145e60 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c013e550 con 0x7f86c006af60 2024-03-29T11:34:33.059 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c013b530 con 0x7f86c00af360 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86c5f4d640 10 monclient(hunting): authenticate will time out at 865.320871s 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bf7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f86c006af60 0x7f86c01da200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43640/0 (socket says 172.21.15.183:43640) 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bf7fe640 1 -- 172.21.15.183:0/2726677425 learned_addr learned my addr 172.21.15.183:0/2726677425 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 10 client.?.objecter ms_handle_connect 0x7f86c006af60 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 10 client.?.objecter ms_handle_connect 0x7f86c00af360 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 10 client.?.objecter ms_handle_connect 0x7f86c0145e60 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791995367 0 0) 0x7f86c013e550 con 0x7f86c006af60 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.048+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a0004eb0 con 0x7f86c006af60 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126297063 0 0) 0x7f86c013e0c0 con 0x7f86c0145e60 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c013e550 con 0x7f86c0145e60 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973896004 0 0) 0x7f86c013b530 con 0x7f86c00af360 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c013e0c0 con 0x7f86c00af360 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2881236475 0 0) 0x7f86a0004eb0 con 0x7f86c006af60 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86c013b530 con 0x7f86c006af60 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f86b0001c60 con 0x7f86c006af60 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): dump: 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.062 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1811430270 0 0) 0x7f86c013e550 con 0x7f86c0145e60 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86a0004eb0 con 0x7f86c0145e60 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f86b8001a90 con 0x7f86c0145e60 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): dump: 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.063 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2324605414 0 0) 0x7f86c013e0c0 con 0x7f86c00af360 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86c013e550 con 0x7f86c00af360 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f86a80018f0 con 0x7f86c00af360 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient(hunting): dump: 2024-03-29T11:34:33.064 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1870487363 0 0) 0x7f86c013b530 con 0x7f86c006af60 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 >> v1:172.21.15.155:6790/0 conn(0x7f86c00af360 legacy=0x7f86c013a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 >> v1:172.21.15.155:6789/0 conn(0x7f86c0145e60 legacy=0x7f86c013ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.065 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 monclient: found mon.b 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c013da30 con 0x7f86c006af60 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86c5f4d640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86c5f4d640 10 monclient: _renew_subs 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86c5f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86c5f4d640 1 -- 172.21.15.183:0/2726677425 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c013dc80 con 0x7f86c006af60 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86c5f4d640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:33.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86c5f4d640 10 monclient: _renew_subs 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86c5f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86c5f4d640 1 -- 172.21.15.183:0/2726677425 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c0071840 con 0x7f86c006af60 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86c5f4d640 1 librados: init done 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86b0003980 con 0x7f86c006af60 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f86b0003d50 con 0x7f86c006af60 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 monclient: dump: 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.067 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86be7fc640 4 set_mon_vals no callback set 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f86b00042c0 con 0x7f86c006af60 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3551128233 0 0) 0x7f86b0005c10 con 0x7f86c006af60 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 10 client.4129.objecter ms_dispatch 0x7f86c0000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.049+0000 7f86bcff9640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.050+0000 7f86c5f4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.050+0000 7f86c5f4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.050+0000 7f86c5f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.050+0000 7f86c5f4d640 1 -- 172.21.15.183:0/2726677425 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c0067c00 con 0x7f86c006af60 2024-03-29T11:34:33.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.052+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f86b0005e70 con 0x7f86c006af60 2024-03-29T11:34:33.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.052+0000 7f86bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.052+0000 7f86bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:33.078 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:33.081+0000 7fa948e10040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:34:33.089 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:33.092+0000 7f20ed843040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:34:33.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.124+0000 7f86c5f4d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a27cee2b-5d5c-421f-bcc5-16b9e3f5b559", "id": 3}] 2024-03-29T11:34:33.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.124+0000 7f86c5f4d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a27cee2b-5d5c-421f-bcc5-16b9e3f5b559", "id": 3}] 2024-03-29T11:34:33.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.124+0000 7f86c5f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.124+0000 7f86c5f4d640 1 -- 172.21.15.183:0/2726677425 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a27cee2b-5d5c-421f-bcc5-16b9e3f5b559", "id": 3} v 0) v1 -- 0x7f86c011dfc0 con 0x7f86c006af60 2024-03-29T11:34:33.126 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.128+0000 7f86bcff9640 1 -- 172.21.15.183:0/2726677425 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a27cee2b-5d5c-421f-bcc5-16b9e3f5b559", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3579305709 0 525046306) 0x7f86b0005850 con 0x7f86c006af60 2024-03-29T11:34:33.126 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.128+0000 7f86bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a27cee2b-5d5c-421f-bcc5-16b9e3f5b559", "id": 3}] 2024-03-29T11:34:33.126 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.128+0000 7f86bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:33.126 INFO:teuthology.orchestra.run.smithi183.stdout:3 2024-03-29T11:34:33.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 10 librados: watch_flush enter 2024-03-29T11:34:33.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 10 librados: watch_flush exit 2024-03-29T11:34:33.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:34:33.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:34:33.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 10 monclient: shutdown 2024-03-29T11:34:33.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 1 -- 172.21.15.183:0/2726677425 >> v1:172.21.15.183:6789/0 conn(0x7f86c006af60 legacy=0x7f86c01da200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 1 -- 172.21.15.183:0/2726677425 shutdown_connections 2024-03-29T11:34:33.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 1 -- 172.21.15.183:0/2726677425 >> 172.21.15.183:0/2726677425 conn(0x7f86c00b58f0 msgr2=0x7f86c00b4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 1 -- 172.21.15.183:0/2726677425 shutdown_connections 2024-03-29T11:34:33.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 1 -- 172.21.15.183:0/2726677425 wait complete. 2024-03-29T11:34:33.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.129+0000 7f86c5f4d640 1 librados: shutdown 2024-03-29T11:34:33.137 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new f4422603-41d7-4505-9ab4-aafb96469ebd 4 2024-03-29T11:34:33.195 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:33.198+0000 7fa948e10040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:34:33.211 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:33.214+0000 7f20ed843040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:34:33.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.239+0000 7fc135c0f640 10 monclient: get_monmap_and_config 2024-03-29T11:34:33.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.239+0000 7fc135c0f640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.239+0000 7fc135c0f640 10 monclient: monmap: 2024-03-29T11:34:33.237 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:33.237 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.237 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:34:33.240994+0000 2024-03-29T11:34:33.238 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:34:33.240994+0000 2024-03-29T11:34:33.238 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.238 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.238 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:33.238 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:33.238 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:33.238 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.246 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.240+0000 7fc135c0f640 1 Processor -- start 2024-03-29T11:34:33.246 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.240+0000 7fc135c0f640 1 -- start start 2024-03-29T11:34:33.246 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.240+0000 7fc135c0f640 10 monclient: init 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fc13009ac40 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 10 monclient(hunting): picked mon.noname-a con 0x7fc13009c870 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 10 monclient(hunting): picked mon.noname-b con 0x7fc1300b4360 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13009dec0 con 0x7fc13009c870 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13009da30 con 0x7fc13009ac40 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1300b39c0 con 0x7fc1300b4360 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 10 monclient(hunting): authenticate will time out at 865.513454s 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12f7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fc13009ac40 0x7fc1300b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43650/0 (socket says 172.21.15.183:43650) 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12f7fe640 1 -- 172.21.15.183:0/1108043924 learned_addr learned my addr 172.21.15.183:0/1108043924 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422757947 0 0) 0x7fc13009da30 con 0x7fc13009ac40 2024-03-29T11:34:33.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc11c003520 con 0x7fc13009ac40 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044784754 0 0) 0x7fc118001110 con 0x7fc13009c870 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc13009da30 con 0x7fc13009c870 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271776726 0 0) 0x7fc1300b39c0 con 0x7fc1300b4360 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc118001110 con 0x7fc1300b4360 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3924844797 0 0) 0x7fc11c003520 con 0x7fc13009ac40 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 >> v1:172.21.15.155:6790/0 conn(0x7fc1300b4360 legacy=0x7fc13014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 >> v1:172.21.15.155:6789/0 conn(0x7fc13009c870 legacy=0x7fc13013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 monclient: found mon.noname-c 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1300b5160 con 0x7fc13009ac40 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.249 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc120001690 con 0x7fc13009ac40 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient: dump: 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.250 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc120001870 con 0x7fc13009ac40 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 1 -- 172.21.15.183:0/1108043924 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc120001c80 con 0x7fc13009ac40 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc135c0f640 4 set_mon_vals no callback set 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.241+0000 7fc12effd640 10 monclient: dump: 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.251 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.242+0000 7fc135c0f640 10 monclient: shutdown 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.242+0000 7fc135c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.242+0000 7fc135c0f640 1 -- 172.21.15.183:0/1108043924 >> v1:172.21.15.183:6789/0 conn(0x7fc13009ac40 legacy=0x7fc1300b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.242+0000 7fc135c0f640 1 -- 172.21.15.183:0/1108043924 shutdown_connections 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.242+0000 7fc135c0f640 1 -- 172.21.15.183:0/1108043924 >> 172.21.15.183:0/1108043924 conn(0x7fc130066b00 msgr2=0x7fc130066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.242+0000 7fc135c0f640 1 -- 172.21.15.183:0/1108043924 shutdown_connections 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.242+0000 7fc135c0f640 1 -- 172.21.15.183:0/1108043924 wait complete. 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.243+0000 7fc135c0f640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.243+0000 7fc135c0f640 10 monclient: monmap: 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.243+0000 7fc135c0f640 1 librados: starting msgr at 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.243+0000 7fc135c0f640 1 librados: starting objecter 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.243+0000 7fc135c0f640 1 Processor -- start 2024-03-29T11:34:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.243+0000 7fc135c0f640 1 -- start start 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.243+0000 7fc135c0f640 1 librados: setting wanted keys 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.243+0000 7fc135c0f640 1 librados: calling monclient init 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.243+0000 7fc135c0f640 10 monclient: init 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fc13009ac40 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 10 monclient(hunting): picked mon.noname-a con 0x7fc1300b1280 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 10 monclient(hunting): picked mon.noname-b con 0x7fc1301d6ed0 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1301422a0 con 0x7fc1300b1280 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc130142f50 con 0x7fc1301d6ed0 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1301ccab0 con 0x7fc13009ac40 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 10 monclient(hunting): authenticate will time out at 865.516410s 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12ffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fc1301d6ed0 0x7fc13013adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43654/0 (socket says 172.21.15.183:43654) 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12ffff640 1 -- 172.21.15.183:0/2957943393 learned_addr learned my addr 172.21.15.183:0/2957943393 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 client.?.objecter ms_handle_connect 0x7fc1301d6ed0 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 client.?.objecter ms_handle_connect 0x7fc1300b1280 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 client.?.objecter ms_handle_connect 0x7fc13009ac40 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993368357 0 0) 0x7fc130142f50 con 0x7fc1301d6ed0 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:34:33.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc110004eb0 con 0x7fc1301d6ed0 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995246790 0 0) 0x7fc1301422a0 con 0x7fc1300b1280 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc130142f50 con 0x7fc1300b1280 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550544416 0 0) 0x7fc1301ccab0 con 0x7fc13009ac40 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1301422a0 con 0x7fc13009ac40 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1014844708 0 0) 0x7fc110004eb0 con 0x7fc1301d6ed0 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1301ccab0 con 0x7fc1301d6ed0 2024-03-29T11:34:33.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc128001c70 con 0x7fc1301d6ed0 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): dump: 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3106780236 0 0) 0x7fc130142f50 con 0x7fc1300b1280 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc110004eb0 con 0x7fc1300b1280 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc1180018a0 con 0x7fc1300b1280 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): dump: 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1060332426 0 0) 0x7fc1301422a0 con 0x7fc13009ac40 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc130142f50 con 0x7fc13009ac40 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc1200019e0 con 0x7fc13009ac40 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient(hunting): dump: 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.259 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 151649405 0 0) 0x7fc1301ccab0 con 0x7fc1301d6ed0 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 >> v1:172.21.15.155:6790/0 conn(0x7fc13009ac40 legacy=0x7fc130141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 >> v1:172.21.15.155:6789/0 conn(0x7fc1300b1280 legacy=0x7fc130142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 monclient: found mon.b 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1301ccc40 con 0x7fc1301d6ed0 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc12cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 10 monclient: _renew_subs 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.244+0000 7fc135c0f640 1 -- 172.21.15.183:0/2957943393 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13013b4d0 con 0x7fc1301d6ed0 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc135c0f640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc135c0f640 10 monclient: _renew_subs 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc135c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc135c0f640 1 -- 172.21.15.183:0/2957943393 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1301e3720 con 0x7fc1301d6ed0 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc135c0f640 1 librados: init done 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc128003990 con 0x7fc1301d6ed0 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc128003da0 con 0x7fc1301d6ed0 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12e7fc640 4 set_mon_vals no callback set 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 10 monclient: dump: 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.262 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc128004350 con 0x7fc1301d6ed0 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3407666257 0 0) 0x7fc128006050 con 0x7fc1301d6ed0 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 10 client.4135.objecter ms_dispatch 0x7fc130000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc12cff9640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc135c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc135c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc135c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.245+0000 7fc135c0f640 1 -- 172.21.15.183:0/2957943393 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc114005080 con 0x7fc1301d6ed0 2024-03-29T11:34:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.247+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc128004600 con 0x7fc1301d6ed0 2024-03-29T11:34:33.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.247+0000 7fc12cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.247+0000 7fc12cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:33.266 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:33.269+0000 7fa948e10040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:33.285 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:33.287+0000 7f20ed843040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:33.312 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:33.315+0000 7fa948e10040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:34:33.317 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.320+0000 7fc135c0f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f4422603-41d7-4505-9ab4-aafb96469ebd", "id": 4}] 2024-03-29T11:34:33.317 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.320+0000 7fc135c0f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f4422603-41d7-4505-9ab4-aafb96469ebd", "id": 4}] 2024-03-29T11:34:33.317 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.320+0000 7fc135c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.317 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.320+0000 7fc135c0f640 1 -- 172.21.15.183:0/2957943393 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f4422603-41d7-4505-9ab4-aafb96469ebd", "id": 4} v 0) v1 -- 0x7fc1140058b0 con 0x7fc1301d6ed0 2024-03-29T11:34:33.321 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.324+0000 7fc12cff9640 1 -- 172.21.15.183:0/2957943393 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f4422603-41d7-4505-9ab4-aafb96469ebd", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 514171412 0 1697012583) 0x7fc128005b40 con 0x7fc1301d6ed0 2024-03-29T11:34:33.321 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.324+0000 7fc12cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f4422603-41d7-4505-9ab4-aafb96469ebd", "id": 4}] 2024-03-29T11:34:33.321 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.324+0000 7fc12cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:33.321 INFO:teuthology.orchestra.run.smithi183.stdout:4 2024-03-29T11:34:33.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.324+0000 7fc135c0f640 10 librados: watch_flush enter 2024-03-29T11:34:33.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.324+0000 7fc135c0f640 10 librados: watch_flush exit 2024-03-29T11:34:33.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.324+0000 7fc135c0f640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T11:34:33.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.324+0000 7fc135c0f640 10 client.4135.objecter successfully canceled tick 2024-03-29T11:34:33.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.324+0000 7fc135c0f640 10 monclient: shutdown 2024-03-29T11:34:33.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.324+0000 7fc135c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.324+0000 7fc135c0f640 1 -- 172.21.15.183:0/2957943393 >> v1:172.21.15.183:6789/0 conn(0x7fc1301d6ed0 legacy=0x7fc13013adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.325+0000 7fc135c0f640 1 -- 172.21.15.183:0/2957943393 shutdown_connections 2024-03-29T11:34:33.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.325+0000 7fc135c0f640 1 -- 172.21.15.183:0/2957943393 >> 172.21.15.183:0/2957943393 conn(0x7fc130066b00 msgr2=0x7fc130066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.323 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.325+0000 7fc135c0f640 1 -- 172.21.15.183:0/2957943393 shutdown_connections 2024-03-29T11:34:33.324 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.325+0000 7fc135c0f640 1 -- 172.21.15.183:0/2957943393 wait complete. 2024-03-29T11:34:33.324 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.325+0000 7fc135c0f640 1 librados: shutdown 2024-03-29T11:34:33.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 283819f5-206d-4020-9122-35eddb22cf6c 5 2024-03-29T11:34:33.333 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:33.336+0000 7f20ed843040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:34:33.366 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:33.369+0000 7fa948e10040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:34:33.389 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:33.392+0000 7f20ed843040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:34:33.413 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:33.417+0000 7fa948e10040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:34:33.433 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.435+0000 7f8678025640 10 monclient: get_monmap_and_config 2024-03-29T11:34:33.433 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.435+0000 7f8678025640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.433 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.435+0000 7f8678025640 10 monclient: monmap: 2024-03-29T11:34:33.433 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:33.433 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.433 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:34:33.436995+0000 2024-03-29T11:34:33.433 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:34:33.436995+0000 2024-03-29T11:34:33.433 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.433 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.434 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:33.434 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:33.434 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:33.434 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.443 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 1 Processor -- start 2024-03-29T11:34:33.443 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 1 -- start start 2024-03-29T11:34:33.443 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 10 monclient: init 2024-03-29T11:34:33.443 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.443 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 10 monclient(hunting): picked mon.noname-b con 0x7f86700b8240 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 10 monclient(hunting): picked mon.noname-a con 0x7f86700b1f60 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 10 monclient(hunting): picked mon.noname-c con 0x7f867009c8c0 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867009bd80 con 0x7f86700b1f60 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867009b960 con 0x7f867009c8c0 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86700b2ec0 con 0x7f86700b8240 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.436+0000 7f8678025640 10 monclient(hunting): authenticate will time out at 865.709164s 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f86765c2640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f867009c8c0 0x7f867014d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43660/0 (socket says 172.21.15.183:43660) 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f86765c2640 1 -- 172.21.15.183:0/2296419479 learned_addr learned my addr 172.21.15.183:0/2296419479 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735839950 0 0) 0x7f867009b960 con 0x7f867009c8c0 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f865c003520 con 0x7f867009c8c0 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500172786 0 0) 0x7f867009bd80 con 0x7f86700b1f60 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f867009b960 con 0x7f86700b1f60 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224580491 0 0) 0x7f86700b2ec0 con 0x7f86700b8240 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.445 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f867009bd80 con 0x7f86700b8240 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1790962955 0 0) 0x7f865c003520 con 0x7f867009c8c0 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 >> v1:172.21.15.155:6790/0 conn(0x7f86700b8240 legacy=0x7f867013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 >> v1:172.21.15.155:6789/0 conn(0x7f86700b1f60 legacy=0x7f867009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 monclient: found mon.noname-c 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867013fb60 con 0x7f867009c8c0 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8668001a10 con 0x7f867009c8c0 2024-03-29T11:34:33.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8678025640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8678025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient: dump: 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8678025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8668001bf0 con 0x7f867009c8c0 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 1 -- 172.21.15.183:0/2296419479 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8668002000 con 0x7f867009c8c0 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8678025640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8678025640 4 set_mon_vals no callback set 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.437+0000 7f8674dbf640 10 monclient: dump: 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.448 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.438+0000 7f8678025640 10 monclient: shutdown 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.438+0000 7f8678025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.438+0000 7f8678025640 1 -- 172.21.15.183:0/2296419479 >> v1:172.21.15.183:6789/0 conn(0x7f867009c8c0 legacy=0x7f867014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.438+0000 7f8678025640 1 -- 172.21.15.183:0/2296419479 shutdown_connections 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.438+0000 7f8678025640 1 -- 172.21.15.183:0/2296419479 >> 172.21.15.183:0/2296419479 conn(0x7f8670066b00 msgr2=0x7f8670066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.438+0000 7f8678025640 1 -- 172.21.15.183:0/2296419479 shutdown_connections 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.438+0000 7f8678025640 1 -- 172.21.15.183:0/2296419479 wait complete. 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 10 monclient: monmap: 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:33.449 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 1 librados: starting msgr at 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 1 librados: starting objecter 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 1 Processor -- start 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 1 -- start start 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 1 librados: setting wanted keys 2024-03-29T11:34:33.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 1 librados: calling monclient init 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 10 monclient: init 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 10 monclient(hunting): picked mon.noname-b con 0x7f867009c8c0 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 10 monclient(hunting): picked mon.noname-c con 0x7f86700b8240 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.439+0000 7f8678025640 10 monclient(hunting): picked mon.noname-a con 0x7f86701d3dc0 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8670136e90 con 0x7f86701d3dc0 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86701d7fe0 con 0x7f867009c8c0 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86701d8170 con 0x7f86700b8240 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 10 monclient(hunting): authenticate will time out at 865.712212s 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8675dc1640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f867009c8c0 0x7f86700b1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43666/0 (socket says 172.21.15.183:43666) 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8675dc1640 1 -- 172.21.15.183:0/28103694 learned_addr learned my addr 172.21.15.183:0/28103694 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:33.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 client.?.objecter ms_handle_connect 0x7f867009c8c0 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 client.?.objecter ms_handle_connect 0x7f86700b8240 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 client.?.objecter ms_handle_connect 0x7f86701d3dc0 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550029825 0 0) 0x7f86701d7fe0 con 0x7f867009c8c0 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8654004eb0 con 0x7f867009c8c0 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241248708 0 0) 0x7f8670136e90 con 0x7f86701d3dc0 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.452 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86701d7fe0 con 0x7f86701d3dc0 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776744335 0 0) 0x7f86701d8170 con 0x7f86700b8240 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8670136e90 con 0x7f86700b8240 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 74218426 0 0) 0x7f8654004eb0 con 0x7f867009c8c0 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86701d8170 con 0x7f867009c8c0 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f866c001970 con 0x7f867009c8c0 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:33.453 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): dump: 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3736956828 0 0) 0x7f86701d7fe0 con 0x7f86701d3dc0 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8654004eb0 con 0x7f86701d3dc0 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8668001d60 con 0x7f86701d3dc0 2024-03-29T11:34:33.454 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): dump: 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3958545039 0 0) 0x7f8670136e90 con 0x7f86700b8240 2024-03-29T11:34:33.455 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86701d7fe0 con 0x7f86700b8240 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8660001870 con 0x7f86700b8240 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient(hunting): dump: 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.456 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 82213141 0 0) 0x7f86701d8170 con 0x7f867009c8c0 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 >> v1:172.21.15.155:6790/0 conn(0x7f86700b8240 legacy=0x7f86701d78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 >> v1:172.21.15.155:6789/0 conn(0x7f86701d3dc0 legacy=0x7f86701d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 monclient: found mon.b 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8670135c30 con 0x7f867009c8c0 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 10 monclient: _renew_subs 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 1 -- 172.21.15.183:0/28103694 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8670135ee0 con 0x7f867009c8c0 2024-03-29T11:34:33.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 10 monclient: _renew_subs 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 1 -- 172.21.15.183:0/28103694 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86701d4cc0 con 0x7f867009c8c0 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8678025640 1 librados: init done 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f866c003610 con 0x7f867009c8c0 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f866c003a20 con 0x7f867009c8c0 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 monclient: dump: 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f8667fff640 4 set_mon_vals no callback set 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f866c003fd0 con 0x7f867009c8c0 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1861331493 0 0) 0x7f866c006810 con 0x7f867009c8c0 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 10 client.4141.objecter ms_dispatch 0x7f8670000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.440+0000 7f86667fc640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.441+0000 7f86667fc640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.441+0000 7f8678025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.441+0000 7f8678025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.459 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.441+0000 7f8678025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.460 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.441+0000 7f8678025640 1 -- 172.21.15.183:0/28103694 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8658005080 con 0x7f867009c8c0 2024-03-29T11:34:33.460 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:33.442+0000 7f20ed843040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:34:33.460 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.443+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f866c004410 con 0x7f867009c8c0 2024-03-29T11:34:33.460 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.443+0000 7f86667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.460 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.443+0000 7f86667fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:33.510 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.513+0000 7f8678025640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "283819f5-206d-4020-9122-35eddb22cf6c", "id": 5}] 2024-03-29T11:34:33.511 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.513+0000 7f8678025640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "283819f5-206d-4020-9122-35eddb22cf6c", "id": 5}] 2024-03-29T11:34:33.511 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.513+0000 7f8678025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.511 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.513+0000 7f8678025640 1 -- 172.21.15.183:0/28103694 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "283819f5-206d-4020-9122-35eddb22cf6c", "id": 5} v 0) v1 -- 0x7f86580058b0 con 0x7f867009c8c0 2024-03-29T11:34:33.514 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.516+0000 7f86667fc640 1 -- 172.21.15.183:0/28103694 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "283819f5-206d-4020-9122-35eddb22cf6c", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2961771578 0 1988412176) 0x7f866c006360 con 0x7f867009c8c0 2024-03-29T11:34:33.514 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.516+0000 7f86667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "283819f5-206d-4020-9122-35eddb22cf6c", "id": 5}] 2024-03-29T11:34:33.514 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.516+0000 7f86667fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:33.514 INFO:teuthology.orchestra.run.smithi183.stdout:5 2024-03-29T11:34:33.515 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 10 librados: watch_flush enter 2024-03-29T11:34:33.515 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 10 librados: watch_flush exit 2024-03-29T11:34:33.516 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:34:33.516 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:34:33.516 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 10 monclient: shutdown 2024-03-29T11:34:33.516 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.516 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 1 -- 172.21.15.183:0/28103694 >> v1:172.21.15.183:6789/0 conn(0x7f867009c8c0 legacy=0x7f86700b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.516 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 1 -- 172.21.15.183:0/28103694 shutdown_connections 2024-03-29T11:34:33.516 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 1 -- 172.21.15.183:0/28103694 >> 172.21.15.183:0/28103694 conn(0x7f8670066b00 msgr2=0x7f86700b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.516 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 1 -- 172.21.15.183:0/28103694 shutdown_connections 2024-03-29T11:34:33.516 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 1 -- 172.21.15.183:0/28103694 wait complete. 2024-03-29T11:34:33.516 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.517+0000 7f8678025640 1 librados: shutdown 2024-03-29T11:34:33.525 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 23b163db-4e0f-46de-b725-5d37debb31e9 6 2024-03-29T11:34:33.625 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.627+0000 7f22b7d82640 10 monclient: get_monmap_and_config 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.627+0000 7f22b7d82640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.627+0000 7f22b7d82640 10 monclient: monmap: 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:34:33.629309+0000 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:34:33.629309+0000 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:33.626 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.627 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.628+0000 7f22b7d82640 1 Processor -- start 2024-03-29T11:34:33.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.628+0000 7f22b7d82640 1 -- start start 2024-03-29T11:34:33.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.628+0000 7f22b7d82640 10 monclient: init 2024-03-29T11:34:33.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:33.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 10 monclient(hunting): picked mon.noname-c con 0x7f22b009ac40 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:33.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 10 monclient(hunting): picked mon.noname-b con 0x7f22b009c870 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:33.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 10 monclient(hunting): picked mon.noname-a con 0x7f22b00b4360 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:33.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b009dec0 con 0x7f22b00b4360 2024-03-29T11:34:33.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b009da30 con 0x7f22b009ac40 2024-03-29T11:34:33.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b00b39c0 con 0x7f22b009c870 2024-03-29T11:34:33.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 10 monclient(hunting): authenticate will time out at 865.901552s 2024-03-29T11:34:33.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b5b1e640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f22b009ac40 0x7f22b00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43672/0 (socket says 172.21.15.183:43672) 2024-03-29T11:34:33.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b5b1e640 1 -- 172.21.15.183:0/322122654 learned_addr learned my addr 172.21.15.183:0/322122654 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:33.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247022727 0 0) 0x7f22b009da30 con 0x7f22b009ac40 2024-03-29T11:34:33.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:34:33.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f229c003520 con 0x7f22b009ac40 2024-03-29T11:34:33.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48506018 0 0) 0x7f22a8001490 con 0x7f22b00b4360 2024-03-29T11:34:33.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b009da30 con 0x7f22b00b4360 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902972793 0 0) 0x7f22b00b39c0 con 0x7f22b009c870 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a8001490 con 0x7f22b009c870 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 63164505 0 0) 0x7f229c003520 con 0x7f22b009ac40 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 >> v1:172.21.15.155:6790/0 conn(0x7f22b009c870 legacy=0x7f22b013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.639 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 >> v1:172.21.15.155:6789/0 conn(0x7f22b00b4360 legacy=0x7f22b014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 monclient: found mon.noname-c 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b00b5160 con 0x7f22b009ac40 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f22a0001690 con 0x7f22b009ac40 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b7d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.640 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.629+0000 7f22b4b1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b4b1c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b4b1c640 10 monclient: dump: 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.641 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b7d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22a0001870 con 0x7f22b009ac40 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b4b1c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b4b1c640 1 -- 172.21.15.183:0/322122654 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f22a0001c80 con 0x7f22b009ac40 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b4b1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b4b1c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b4b1c640 10 monclient: dump: 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.642 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b7d82640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b7d82640 4 set_mon_vals no callback set 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b7d82640 10 monclient: shutdown 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b7d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b7d82640 1 -- 172.21.15.183:0/322122654 >> v1:172.21.15.183:6789/0 conn(0x7f22b009ac40 legacy=0x7f22b00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b7d82640 1 -- 172.21.15.183:0/322122654 shutdown_connections 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b7d82640 1 -- 172.21.15.183:0/322122654 >> 172.21.15.183:0/322122654 conn(0x7f22b0066b00 msgr2=0x7f22b0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.630+0000 7f22b7d82640 1 -- 172.21.15.183:0/322122654 shutdown_connections 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.631+0000 7f22b7d82640 1 -- 172.21.15.183:0/322122654 wait complete. 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.631+0000 7f22b7d82640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.631+0000 7f22b7d82640 10 monclient: monmap: 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:33.644 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.632+0000 7f22b7d82640 1 librados: starting msgr at 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.632+0000 7f22b7d82640 1 librados: starting objecter 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.632+0000 7f22b7d82640 1 Processor -- start 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.632+0000 7f22b7d82640 1 -- start start 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.632+0000 7f22b7d82640 1 librados: setting wanted keys 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.632+0000 7f22b7d82640 1 librados: calling monclient init 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.632+0000 7f22b7d82640 10 monclient: init 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b7d82640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b7d82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b7d82640 10 monclient(hunting): picked mon.noname-b con 0x7f22b009ac40 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b7d82640 10 monclient(hunting): picked mon.noname-c con 0x7f22b00b1860 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:33.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b7d82640 10 monclient(hunting): picked mon.noname-a con 0x7f22b013b5b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b7d82640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b01d7660 con 0x7f22b013b5b0 2024-03-29T11:34:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b7d82640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b00ade20 con 0x7f22b009ac40 2024-03-29T11:34:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b7d82640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b01406b0 con 0x7f22b00b1860 2024-03-29T11:34:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b7d82640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b7d82640 10 monclient(hunting): authenticate will time out at 865.905575s 2024-03-29T11:34:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b5b1e640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f22b009ac40 0x7f22b013ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43688/0 (socket says 172.21.15.183:43688) 2024-03-29T11:34:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22b5b1e640 1 -- 172.21.15.183:0/4260527247 learned_addr learned my addr 172.21.15.183:0/4260527247 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 client.?.objecter ms_handle_connect 0x7f22b009ac40 2024-03-29T11:34:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258423420 0 0) 0x7f22b00ade20 con 0x7f22b009ac40 2024-03-29T11:34:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2294004eb0 con 0x7f22b009ac40 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 client.?.objecter ms_handle_connect 0x7f22b013b5b0 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 client.?.objecter ms_handle_connect 0x7f22b00b1860 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295213117 0 0) 0x7f22b01d7660 con 0x7f22b013b5b0 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b00ade20 con 0x7f22b013b5b0 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791010588 0 0) 0x7f22b01406b0 con 0x7f22b00b1860 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b01d7660 con 0x7f22b00b1860 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1551790721 0 0) 0x7f2294004eb0 con 0x7f22b009ac40 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22b01406b0 con 0x7f22b009ac40 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f22a0001bc0 con 0x7f22b009ac40 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): dump: 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1654959792 0 0) 0x7f22b00ade20 con 0x7f22b013b5b0 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2294004eb0 con 0x7f22b013b5b0 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f22a8001a80 con 0x7f22b013b5b0 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.633+0000 7f22a67fc640 10 monclient(hunting): dump: 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1558194583 0 0) 0x7f22b01d7660 con 0x7f22b00b1860 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22b00ade20 con 0x7f22b00b1860 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f22ac001820 con 0x7f22b00b1860 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 monclient(hunting): dump: 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2227852047 0 0) 0x7f22b01406b0 con 0x7f22b009ac40 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 >> v1:172.21.15.155:6790/0 conn(0x7f22b00b1860 legacy=0x7f22b013aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 >> v1:172.21.15.155:6789/0 conn(0x7f22b013b5b0 legacy=0x7f22b01e2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 monclient: found mon.b 2024-03-29T11:34:33.652 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b01cccd0 con 0x7f22b009ac40 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22b7d82640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22b7d82640 10 monclient: _renew_subs 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22b7d82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22b7d82640 1 -- 172.21.15.183:0/4260527247 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b01cce60 con 0x7f22b009ac40 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22b7d82640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22b7d82640 10 monclient: _renew_subs 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22b7d82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.653 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22b7d82640 1 -- 172.21.15.183:0/4260527247 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b01d7190 con 0x7f22b009ac40 2024-03-29T11:34:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22b7d82640 1 librados: init done 2024-03-29T11:34:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22a0001170 con 0x7f22b009ac40 2024-03-29T11:34:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f22a0004d90 con 0x7f22b009ac40 2024-03-29T11:34:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 monclient: dump: 2024-03-29T11:34:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a7fff640 4 set_mon_vals no callback set 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f22a0005300 con 0x7f22b009ac40 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1327882156 0 0) 0x7f22a0006b20 con 0x7f22b009ac40 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 10 client.4147.objecter ms_dispatch 0x7f22b0000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:34:33.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.634+0000 7f22a67fc640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:34:33.656 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.635+0000 7f22b7d82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.656 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.635+0000 7f22b7d82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.656 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.635+0000 7f22b7d82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.656 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.635+0000 7f22b7d82640 1 -- 172.21.15.183:0/4260527247 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2298005080 con 0x7f22b009ac40 2024-03-29T11:34:33.656 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.637+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f22a00065e0 con 0x7f22b009ac40 2024-03-29T11:34:33.656 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.637+0000 7f22a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.656 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.637+0000 7f22a67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:33.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.710+0000 7f22b7d82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "23b163db-4e0f-46de-b725-5d37debb31e9", "id": 6}] 2024-03-29T11:34:33.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.710+0000 7f22b7d82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "23b163db-4e0f-46de-b725-5d37debb31e9", "id": 6}] 2024-03-29T11:34:33.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.710+0000 7f22b7d82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.710+0000 7f22b7d82640 1 -- 172.21.15.183:0/4260527247 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "23b163db-4e0f-46de-b725-5d37debb31e9", "id": 6} v 0) v1 -- 0x7f22980058b0 con 0x7f22b009ac40 2024-03-29T11:34:33.711 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.714+0000 7f22a67fc640 1 -- 172.21.15.183:0/4260527247 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "23b163db-4e0f-46de-b725-5d37debb31e9", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2592693035 0 1113811849) 0x7f22a0006850 con 0x7f22b009ac40 2024-03-29T11:34:33.712 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.714+0000 7f22a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "23b163db-4e0f-46de-b725-5d37debb31e9", "id": 6}] 2024-03-29T11:34:33.712 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.714+0000 7f22a67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:33.712 INFO:teuthology.orchestra.run.smithi183.stdout:6 2024-03-29T11:34:33.713 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 10 librados: watch_flush enter 2024-03-29T11:34:33.713 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 10 librados: watch_flush exit 2024-03-29T11:34:33.713 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:34:33.713 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:34:33.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 10 monclient: shutdown 2024-03-29T11:34:33.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 1 -- 172.21.15.183:0/4260527247 >> v1:172.21.15.183:6789/0 conn(0x7f22b009ac40 legacy=0x7f22b013ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 1 -- 172.21.15.183:0/4260527247 shutdown_connections 2024-03-29T11:34:33.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 1 -- 172.21.15.183:0/4260527247 >> 172.21.15.183:0/4260527247 conn(0x7f22b0066b00 msgr2=0x7f22b013cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 1 -- 172.21.15.183:0/4260527247 shutdown_connections 2024-03-29T11:34:33.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 1 -- 172.21.15.183:0/4260527247 wait complete. 2024-03-29T11:34:33.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.715+0000 7f22b7d82640 1 librados: shutdown 2024-03-29T11:34:33.723 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new fe64015c-477d-4664-a96a-bc5ef201be2d 7 2024-03-29T11:34:33.801 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:33.805+0000 7fa948e10040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:34:33.823 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.825+0000 7f574219d640 10 monclient: get_monmap_and_config 2024-03-29T11:34:33.823 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.825+0000 7f574219d640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.823 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.825+0000 7f574219d640 10 monclient: monmap: 2024-03-29T11:34:33.823 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:33.823 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.823 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:34:33.826743+0000 2024-03-29T11:34:33.823 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:34:33.826743+0000 2024-03-29T11:34:33.823 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.824 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.824 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:33.824 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:33.824 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:33.824 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 1 Processor -- start 2024-03-29T11:34:33.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 1 -- start start 2024-03-29T11:34:33.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 10 monclient: init 2024-03-29T11:34:33.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:33.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 10 monclient(hunting): picked mon.noname-b con 0x7f573c09ac40 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:33.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 10 monclient(hunting): picked mon.noname-c con 0x7f573c09c870 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:33.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 10 monclient(hunting): picked mon.noname-a con 0x7f573c0b4360 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:33.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c09dec0 con 0x7f573c0b4360 2024-03-29T11:34:33.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c09da30 con 0x7f573c09c870 2024-03-29T11:34:33.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c0b39c0 con 0x7f573c09ac40 2024-03-29T11:34:33.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f574219d640 10 monclient(hunting): authenticate will time out at 866.099004s 2024-03-29T11:34:33.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f573affd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f573c09c870 0x7f573c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43704/0 (socket says 172.21.15.183:43704) 2024-03-29T11:34:33.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.826+0000 7f573affd640 1 -- 172.21.15.183:0/3866485126 learned_addr learned my addr 172.21.15.183:0/3866485126 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635037242 0 0) 0x7f573c09da30 con 0x7f573c09c870 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5724003520 con 0x7f573c09c870 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620925364 0 0) 0x7f573c0b39c0 con 0x7f573c09ac40 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:34:33.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c09da30 con 0x7f573c09ac40 2024-03-29T11:34:33.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021915183 0 0) 0x7f5734001490 con 0x7f573c0b4360 2024-03-29T11:34:33.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:34:33.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c0b39c0 con 0x7f573c0b4360 2024-03-29T11:34:33.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 813413393 0 0) 0x7f5724003520 con 0x7f573c09c870 2024-03-29T11:34:33.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 >> v1:172.21.15.155:6790/0 conn(0x7f573c09ac40 legacy=0x7f573c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 >> v1:172.21.15.155:6789/0 conn(0x7f573c0b4360 legacy=0x7f573c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:34:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:34:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0b5160 con 0x7f573c09c870 2024-03-29T11:34:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f574219d640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:34:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f5730001690 con 0x7f573c09c870 2024-03-29T11:34:33.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f574219d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:33.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient: dump: 2024-03-29T11:34:33.839 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.839 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.839 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.839 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.839 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.839 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.840 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.840 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.840 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.840 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f574219d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5730001870 con 0x7f573c09c870 2024-03-29T11:34:33.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 1 -- 172.21.15.183:0/3866485126 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f5730001c80 con 0x7f573c09c870 2024-03-29T11:34:33.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f574219d640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:33.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f574219d640 4 set_mon_vals no callback set 2024-03-29T11:34:33.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:33.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.827+0000 7f573a7fc640 10 monclient: dump: 2024-03-29T11:34:33.841 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.841 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.841 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.841 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.841 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.841 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.841 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.842 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.842 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.842 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.828+0000 7f574219d640 10 monclient: shutdown 2024-03-29T11:34:33.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.828+0000 7f574219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.828+0000 7f574219d640 1 -- 172.21.15.183:0/3866485126 >> v1:172.21.15.183:6789/0 conn(0x7f573c09c870 legacy=0x7f573c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.828+0000 7f574219d640 1 -- 172.21.15.183:0/3866485126 shutdown_connections 2024-03-29T11:34:33.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.828+0000 7f574219d640 1 -- 172.21.15.183:0/3866485126 >> 172.21.15.183:0/3866485126 conn(0x7f573c066b00 msgr2=0x7f573c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.828+0000 7f574219d640 1 -- 172.21.15.183:0/3866485126 shutdown_connections 2024-03-29T11:34:33.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.828+0000 7f574219d640 1 -- 172.21.15.183:0/3866485126 wait complete. 2024-03-29T11:34:33.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.828+0000 7f574219d640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.828+0000 7f574219d640 10 monclient: monmap: 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.843 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 1 librados: starting msgr at 2024-03-29T11:34:33.844 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 1 librados: starting objecter 2024-03-29T11:34:33.844 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 1 Processor -- start 2024-03-29T11:34:33.844 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 1 -- start start 2024-03-29T11:34:33.844 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 1 librados: setting wanted keys 2024-03-29T11:34:33.844 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 1 librados: calling monclient init 2024-03-29T11:34:33.844 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 10 monclient: init 2024-03-29T11:34:33.844 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.844 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:33.844 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 10 monclient(hunting): picked mon.noname-b con 0x7f573c09ac40 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:33.844 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 10 monclient(hunting): picked mon.noname-c con 0x7f573c140660 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:33.844 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 10 monclient(hunting): picked mon.noname-a con 0x7f573c140f80 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c1423a0 con 0x7f573c140f80 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c140140 con 0x7f573c09ac40 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c0b8040 con 0x7f573c140660 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.829+0000 7f574219d640 10 monclient(hunting): authenticate will time out at 866.102105s 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f573b7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f573c09ac40 0x7f573c1d7480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:43712/0 (socket says 172.21.15.183:43712) 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f573b7fe640 1 -- 172.21.15.183:0/1798475513 learned_addr learned my addr 172.21.15.183:0/1798475513 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 client.?.objecter ms_handle_connect 0x7f573c09ac40 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 client.?.objecter ms_handle_connect 0x7f573c140f80 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 client.?.objecter ms_handle_connect 0x7f573c140660 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443307950 0 0) 0x7f573c140140 con 0x7f573c09ac40 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5728004eb0 con 0x7f573c09ac40 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450162792 0 0) 0x7f573c1423a0 con 0x7f573c140f80 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c140140 con 0x7f573c140f80 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101109665 0 0) 0x7f573c0b8040 con 0x7f573c140660 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c1423a0 con 0x7f573c140660 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 506405712 0 0) 0x7f573c140140 con 0x7f573c140f80 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f573c0b8040 con 0x7f573c140f80 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f5734001bc0 con 0x7f573c140f80 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): dump: 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 351847863 0 0) 0x7f5728004eb0 con 0x7f573c09ac40 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f573c140140 con 0x7f573c09ac40 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2551263061 0 0) 0x7f573c1423a0 con 0x7f573c140660 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5728004eb0 con 0x7f573c140660 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f572c001930 con 0x7f573c09ac40 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): dump: 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.849 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f5730001b80 con 0x7f573c140660 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient(hunting): dump: 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.850 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 25674311 0 0) 0x7f573c0b8040 con 0x7f573c140f80 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 >> v1:172.21.15.155:6790/0 conn(0x7f573c140660 legacy=0x7f573c0b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 >> v1:172.21.15.183:6789/0 conn(0x7f573c09ac40 legacy=0x7f573c1d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 monclient: found mon.a 2024-03-29T11:34:33.851 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0b81d0 con 0x7f573c140f80 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f574219d640 5 monclient: authenticate success, global_id 4158 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f574219d640 10 monclient: _renew_subs 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f574219d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f574219d640 1 -- 172.21.15.183:0/1798475513 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c1d7b90 con 0x7f573c140f80 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f574219d640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f574219d640 10 monclient: _renew_subs 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f574219d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:33.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f574219d640 1 -- 172.21.15.183:0/1798475513 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c0b19f0 con 0x7f573c140f80 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f574219d640 1 librados: init done 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5734001fd0 con 0x7f573c140f80 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f5734003b30 con 0x7f573c140f80 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 monclient: dump: 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:33.853 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5739ffb640 4 set_mon_vals no callback set 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5734004020 con 0x7f573c140f80 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2747862983 0 0) 0x7f5734004930 con 0x7f573c140f80 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 10 client.4158.objecter ms_dispatch 0x7f573c000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 3 client.4158.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.830+0000 7f5723fff640 3 client.4158.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:34:33.854 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.831+0000 7f5723fff640 20 client.4158.objecter dump_active .. 0 homeless 2024-03-29T11:34:33.855 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.831+0000 7f574219d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.855 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.831+0000 7f574219d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.855 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.831+0000 7f574219d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:33.855 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.831+0000 7f574219d640 1 -- 172.21.15.183:0/1798475513 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c13ffa0 con 0x7f573c140f80 2024-03-29T11:34:33.855 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.833+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f573c13ffa0 con 0x7f573c140f80 2024-03-29T11:34:33.855 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.833+0000 7f5723fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.855 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.833+0000 7f5723fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:33.856 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:33.842+0000 7f20ed843040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:34:33.902 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.904+0000 7f574219d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fe64015c-477d-4664-a96a-bc5ef201be2d", "id": 7}] 2024-03-29T11:34:33.902 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.904+0000 7f574219d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fe64015c-477d-4664-a96a-bc5ef201be2d", "id": 7}] 2024-03-29T11:34:33.902 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.904+0000 7f574219d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:33.902 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.904+0000 7f574219d640 1 -- 172.21.15.183:0/1798475513 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fe64015c-477d-4664-a96a-bc5ef201be2d", "id": 7} v 0) v1 -- 0x7f573c1402d0 con 0x7f573c140f80 2024-03-29T11:34:33.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.906+0000 7f5723fff640 1 -- 172.21.15.183:0/1798475513 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fe64015c-477d-4664-a96a-bc5ef201be2d", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 279593429 0 1371665406) 0x7f57340042d0 con 0x7f573c140f80 2024-03-29T11:34:33.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.906+0000 7f5723fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fe64015c-477d-4664-a96a-bc5ef201be2d", "id": 7}] 2024-03-29T11:34:33.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.906+0000 7f5723fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:33.904 INFO:teuthology.orchestra.run.smithi183.stdout:7 2024-03-29T11:34:33.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 10 librados: watch_flush enter 2024-03-29T11:34:33.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 10 librados: watch_flush exit 2024-03-29T11:34:33.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 20 client.4158.objecter shutdown clearing up homeless session... 2024-03-29T11:34:33.905 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 10 client.4158.objecter successfully canceled tick 2024-03-29T11:34:33.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 10 monclient: shutdown 2024-03-29T11:34:33.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 1 -- 172.21.15.183:0/1798475513 >> v1:172.21.15.155:6789/0 conn(0x7f573c140f80 legacy=0x7f573c1e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 1 -- 172.21.15.183:0/1798475513 shutdown_connections 2024-03-29T11:34:33.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 1 -- 172.21.15.183:0/1798475513 >> 172.21.15.183:0/1798475513 conn(0x7f573c066b00 msgr2=0x7f573c13c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 1 -- 172.21.15.183:0/1798475513 shutdown_connections 2024-03-29T11:34:33.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 1 -- 172.21.15.183:0/1798475513 wait complete. 2024-03-29T11:34:33.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T11:34:33.907+0000 7f574219d640 1 librados: shutdown 2024-03-29T11:34:33.915 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:34:33.915 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:34:33.918 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:34:33.918 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:34:33.918 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:34:33.920 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:34:33.920 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:34:33.920 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:34:33.922 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:34:33.922 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:34:33.923 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:34:33.924 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:34:33.925 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:34:33.925 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:34:33.927 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:34:33.927 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:34:33.927 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:34:33.929 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:34:33.929 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:34:33.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:34:33.931 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:34:33.932 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:34:33.932 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:34:33.934 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:34:33.934 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:33.942 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:33.945+0000 7fa948e10040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:34:34.005 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:34.008+0000 7f20ed843040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:34:34.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.065+0000 7f774791c640 10 monclient: get_monmap_and_config 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.065+0000 7f774791c640 10 monclient: build_initial_monmap 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.065+0000 7f774791c640 10 monclient: monmap: 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T11:34:34.065522+0000 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T11:34:34.065522+0000 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:34.062 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:34.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.065+0000 7f774791c640 1 Processor -- start 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.065+0000 7f774791c640 1 -- start start 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.065+0000 7f774791c640 10 monclient: init 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f774791c640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f774791c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f774791c640 10 monclient(hunting): picked mon.noname-c con 0x7f77400b8240 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f774791c640 10 monclient(hunting): picked mon.noname-a con 0x7f77400b1f60 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f774791c640 10 monclient(hunting): picked mon.noname-b con 0x7f774009c8c0 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f774791c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774009bd80 con 0x7f77400b1f60 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f774791c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774009b960 con 0x7f77400b8240 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f774791c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77400b2ec0 con 0x7f774009c8c0 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f774791c640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f774791c640 10 monclient(hunting): authenticate will time out at 856.192578s 2024-03-29T11:34:34.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f7745eb9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f774009c8c0 0x7f774014d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55202/0 (socket says 172.21.15.155:55202) 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f7745eb9640 1 -- 172.21.15.155:0/2348940318 learned_addr learned my addr 172.21.15.155:0/2348940318 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.066+0000 7f7744eb7640 1 --1- 172.21.15.155:0/2348940318 >> v1:172.21.15.155:6789/0 conn(0x7f77400b1f60 0x7f774009c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36262/0 (socket says 172.21.15.155:36262) 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 -- 172.21.15.155:0/2348940318 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518758326 0 0) 0x7f77400b2ec0 con 0x7f774009c8c0 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient(hunting): my global_id is 4178 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 -- 172.21.15.155:0/2348940318 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f772c003520 con 0x7f774009c8c0 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 -- 172.21.15.155:0/2348940318 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139220655 0 0) 0x7f774009bd80 con 0x7f77400b1f60 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 -- 172.21.15.155:0/2348940318 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77400b2ec0 con 0x7f77400b1f60 2024-03-29T11:34:34.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 -- 172.21.15.155:0/2348940318 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1295491376 0 0) 0x7f77400b2ec0 con 0x7f77400b1f60 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 -- 172.21.15.155:0/2348940318 >> v1:172.21.15.155:6790/0 conn(0x7f774009c8c0 legacy=0x7f774014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 -- 172.21.15.155:0/2348940318 >> v1:172.21.15.183:6789/0 conn(0x7f77400b8240 legacy=0x7f774013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 monclient: found mon.noname-a 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 -- 172.21.15.155:0/2348940318 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f774013fb60 con 0x7f77400b1f60 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 -- 172.21.15.155:0/2348940318 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f773c001670 con 0x7f77400b1f60 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient: dump: 2024-03-29T11:34:34.073 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f774791c640 5 monclient: authenticate success, global_id 4185 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f774791c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 -- 172.21.15.155:0/2348940318 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f773c0017d0 con 0x7f77400b1f60 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:34.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 1 -- 172.21.15.155:0/2348940318 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f773c001c20 con 0x7f77400b1f60 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f774791c640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f774791c640 4 set_mon_vals no callback set 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f7737fff640 10 monclient: dump: 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:34.075 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f774791c640 10 monclient: shutdown 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f774791c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.067+0000 7f774791c640 1 -- 172.21.15.155:0/2348940318 >> v1:172.21.15.155:6789/0 conn(0x7f77400b1f60 legacy=0x7f774009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.068+0000 7f774791c640 1 -- 172.21.15.155:0/2348940318 shutdown_connections 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.068+0000 7f774791c640 1 -- 172.21.15.155:0/2348940318 >> 172.21.15.155:0/2348940318 conn(0x7f7740066b00 msgr2=0x7f7740066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.068+0000 7f774791c640 1 -- 172.21.15.155:0/2348940318 shutdown_connections 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.068+0000 7f774791c640 1 -- 172.21.15.155:0/2348940318 wait complete. 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.068+0000 7f774791c640 10 monclient: build_initial_monmap 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.068+0000 7f774791c640 10 monclient: monmap: 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:34.076 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.068+0000 7f774791c640 1 librados: starting msgr at 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.068+0000 7f774791c640 1 librados: starting objecter 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 1 Processor -- start 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 1 -- start start 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 1 librados: setting wanted keys 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 1 librados: calling monclient init 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 10 monclient: init 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:34.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 10 monclient(hunting): picked mon.noname-c con 0x7f774009c8c0 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 10 monclient(hunting): picked mon.noname-b con 0x7f77400b8240 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 10 monclient(hunting): picked mon.noname-a con 0x7f77401315e0 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774006f840 con 0x7f77401315e0 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f77456b8640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f774009c8c0 0x7f7740070220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55206/0 (socket says 172.21.15.155:55206) 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f77456b8640 1 -- 172.21.15.155:0/1095338105 learned_addr learned my addr 172.21.15.155:0/1095338105 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f7735ffb640 10 client.?.objecter ms_handle_connect 0x7f77401315e0 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f7735ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f7735ffb640 10 client.?.objecter ms_handle_connect 0x7f774009c8c0 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f7735ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774006fb20 con 0x7f77400b8240 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774006fe00 con 0x7f774009c8c0 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f774791c640 10 monclient(hunting): authenticate will time out at 856.195764s 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824169643 0 0) 0x7f774006f840 con 0x7f77401315e0 2024-03-29T11:34:34.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f7735ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f7735ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f7735ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.069+0000 7f7735ffb640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7724004eb0 con 0x7f77401315e0 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554925349 0 0) 0x7f774006fe00 con 0x7f774009c8c0 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f774006f840 con 0x7f774009c8c0 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 client.?.objecter ms_handle_connect 0x7f77400b8240 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112413722 0 0) 0x7f774006fb20 con 0x7f77400b8240 2024-03-29T11:34:34.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f774006fe00 con 0x7f77400b8240 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1635229025 0 0) 0x7f7724004eb0 con 0x7f77401315e0 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f774006fb20 con 0x7f77401315e0 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 987773174 0 0) 0x7f774006f840 con 0x7f774009c8c0 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7724004eb0 con 0x7f774009c8c0 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7738001bb0 con 0x7f77401315e0 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): dump: 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:34.080 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7730001870 con 0x7f774009c8c0 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient(hunting): dump: 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:34.081 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1036331047 0 0) 0x7f774006fb20 con 0x7f77401315e0 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 >> v1:172.21.15.155:6790/0 conn(0x7f774009c8c0 legacy=0x7f7740070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 >> v1:172.21.15.183:6789/0 conn(0x7f77400b8240 legacy=0x7f774006e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 monclient: found mon.a 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:34.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7740070ed0 con 0x7f77401315e0 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f774791c640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f774791c640 10 monclient: _renew_subs 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7738004b60 con 0x7f77401315e0 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f774791c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f774791c640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7740144650 con 0x7f77401315e0 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7738001d10 con 0x7f77401315e0 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 10 monclient: dump: 2024-03-29T11:34:34.083 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f77377fe640 4 set_mon_vals no callback set 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f774791c640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f774791c640 10 monclient: _renew_subs 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7738005200 con 0x7f77401315e0 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f774791c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:34.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f774791c640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7740132820 con 0x7f77401315e0 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.070+0000 7f774791c640 1 librados: init done 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.071+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f7738005ce0 con 0x7f77401315e0 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.071+0000 7f7735ffb640 10 client.4188.objecter ms_dispatch 0x7f7740000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.071+0000 7f7735ffb640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.071+0000 7f7735ffb640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.071+0000 7f7735ffb640 20 client.4188.objecter dump_active .. 0 homeless 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.071+0000 7f774791c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.071+0000 7f774791c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.071+0000 7f774791c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.071+0000 7f774791c640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f774009b170 con 0x7f77401315e0 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.073+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7738005730 con 0x7f77401315e0 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.073+0000 7f7735ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:34.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.073+0000 7f7735ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:34.090 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:34.093+0000 7fa948e10040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:34:34.128 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:34.130+0000 7f20ed843040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:34:34.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.135+0000 7f774791c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:34.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.135+0000 7f774791c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:34.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.135+0000 7f774791c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:34.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.135+0000 7f774791c640 1 -- 172.21.15.155:0/1095338105 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f774004f730 con 0x7f77401315e0 2024-03-29T11:34:34.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.135+0000 7f7735ffb640 1 -- 172.21.15.155:0/1095338105 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f7738005950 con 0x7f77401315e0 2024-03-29T11:34:34.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.135+0000 7f7735ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:34.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.135+0000 7f7735ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:34.132 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T11:34:34.132 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:34.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.136+0000 7f774791c640 10 librados: watch_flush enter 2024-03-29T11:34:34.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.136+0000 7f774791c640 10 librados: watch_flush exit 2024-03-29T11:34:34.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.136+0000 7f774791c640 20 client.4188.objecter shutdown clearing up homeless session... 2024-03-29T11:34:34.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.136+0000 7f774791c640 10 client.4188.objecter successfully canceled tick 2024-03-29T11:34:34.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.136+0000 7f774791c640 10 monclient: shutdown 2024-03-29T11:34:34.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.136+0000 7f774791c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:34.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.136+0000 7f774791c640 1 -- 172.21.15.155:0/1095338105 >> v1:172.21.15.155:6789/0 conn(0x7f77401315e0 legacy=0x7f77401e65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:34.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.136+0000 7f774791c640 1 -- 172.21.15.155:0/1095338105 shutdown_connections 2024-03-29T11:34:34.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.136+0000 7f774791c640 1 -- 172.21.15.155:0/1095338105 >> 172.21.15.155:0/1095338105 conn(0x7f7740066b00 msgr2=0x7f7740069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:34.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.136+0000 7f774791c640 1 -- 172.21.15.155:0/1095338105 shutdown_connections 2024-03-29T11:34:34.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.136+0000 7f774791c640 1 -- 172.21.15.155:0/1095338105 wait complete. 2024-03-29T11:34:34.135 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T11:34:34.137+0000 7f774791c640 1 librados: shutdown 2024-03-29T11:34:34.137 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:34.141+0000 7fa948e10040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:34:34.144 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:34:34.144 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:34:34.173 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:34.175+0000 7f20ed843040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:34:34.252 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:34.256+0000 7fa948e10040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:34.286 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:34.288+0000 7f20ed843040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:34.349 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:34.352+0000 7fa948e10040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:34:34.381 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:34.384+0000 7f20ed843040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:34:34.504 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:34.507+0000 7fa948e10040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:34.523 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-03-29T11:34:34.526+0000 7f986d619640 -1 Falling back to public interface 2024-03-29T11:34:34.532 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-29T11:34:34.534+0000 7f05df4aa640 -1 Falling back to public interface 2024-03-29T11:34:34.537 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:34.539+0000 7f20ed843040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:34.537 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-29T11:34:34.539+0000 7f65c9c0d640 -1 Falling back to public interface 2024-03-29T11:34:34.540 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-29T11:34:34.542+0000 7fb5bcd91640 -1 Falling back to public interface 2024-03-29T11:34:34.544 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-29T11:34:34.546+0000 7f42c4cb1640 -1 Falling back to public interface 2024-03-29T11:34:34.544 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-03-29T11:34:34.547+0000 7f4e90706640 -1 Falling back to public interface 2024-03-29T11:34:34.547 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:34.551+0000 7fa948e10040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:34.549 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-03-29T11:34:34.552+0000 7fa02c21c640 -1 Falling back to public interface 2024-03-29T11:34:34.555 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-03-29T11:34:34.558+0000 7fc8ea906640 -1 Falling back to public interface 2024-03-29T11:34:34.583 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:34.585+0000 7f20ed843040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:34.702 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:34.706+0000 7fa948e10040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:34:34.737 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:34.739+0000 7f20ed843040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:34:34.950 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:35.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.042+0000 7f7bacd0e640 10 monclient: get_monmap_and_config 2024-03-29T11:34:35.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.042+0000 7f7bacd0e640 10 monclient: build_initial_monmap 2024-03-29T11:34:35.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.042+0000 7f7bacd0e640 10 monclient: monmap: 2024-03-29T11:34:35.039 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:34:35.039 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:35.039 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:34:35.042848+0000 2024-03-29T11:34:35.039 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:34:35.042848+0000 2024-03-29T11:34:35.039 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:35.039 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:35.039 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:35.040 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:35.040 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:35.040 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:35.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.042+0000 7f7bacd0e640 1 Processor -- start 2024-03-29T11:34:35.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 1 -- start start 2024-03-29T11:34:35.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 10 monclient: init 2024-03-29T11:34:35.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:35.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:35.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 10 monclient(hunting): picked mon.noname-b con 0x7f7ba81457f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:35.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 10 monclient(hunting): picked mon.noname-c con 0x7f7ba813f650 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:35.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 10 monclient(hunting): picked mon.noname-a con 0x7f7ba80a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:35.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba80aaef0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba8145bd0 con 0x7f7ba813f650 2024-03-29T11:34:35.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba8145eb0 con 0x7f7ba81457f0 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7bacd0e640 10 monclient(hunting): authenticate will time out at 857.169695s 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7ba6d9d640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7ba80a9d70 0x7f7ba814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36280/0 (socket says 172.21.15.155:36280) 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7ba6d9d640 1 -- 172.21.15.155:0/2577251008 learned_addr learned my addr 172.21.15.155:0/2577251008 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.043+0000 7f7ba659c640 1 --1- 172.21.15.155:0/2577251008 >> v1:172.21.15.155:6790/0 conn(0x7f7ba81457f0 0x7f7ba813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55212/0 (socket says 172.21.15.155:55212) 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970541126 0 0) 0x7f7ba8145eb0 con 0x7f7ba81457f0 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b90003520 con 0x7f7ba81457f0 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666025664 0 0) 0x7f7ba80aaef0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba8145eb0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411346118 0 0) 0x7f7ba8145bd0 con 0x7f7ba813f650 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba80aaef0 con 0x7f7ba813f650 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 313463975 0 0) 0x7f7ba8145eb0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 >> v1:172.21.15.155:6790/0 conn(0x7f7ba81457f0 legacy=0x7f7ba813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 >> v1:172.21.15.183:6789/0 conn(0x7f7ba813f650 legacy=0x7f7ba80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient: _finish_hunting 0 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 monclient: found mon.noname-a 2024-03-29T11:34:35.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba8146340 con 0x7f7ba80a9d70 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient: _finish_auth 0 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient: _check_auth_tickets 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7ba00019f0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7bacd0e640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient: dump: 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.051 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7bacd0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ba0001b50 con 0x7f7ba80a9d70 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 1 -- 172.21.15.155:0/2577251008 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7ba0001fa0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7bacd0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7bacd0e640 4 set_mon_vals no callback set 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.044+0000 7f7ba559a640 10 monclient: dump: 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:35.052 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.045+0000 7f7bacd0e640 10 monclient: shutdown 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.045+0000 7f7bacd0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.045+0000 7f7bacd0e640 1 -- 172.21.15.155:0/2577251008 >> v1:172.21.15.155:6789/0 conn(0x7f7ba80a9d70 legacy=0x7f7ba814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.045+0000 7f7bacd0e640 1 -- 172.21.15.155:0/2577251008 shutdown_connections 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.045+0000 7f7bacd0e640 1 -- 172.21.15.155:0/2577251008 >> 172.21.15.155:0/2577251008 conn(0x7f7ba8066a00 msgr2=0x7f7ba8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.045+0000 7f7bacd0e640 1 -- 172.21.15.155:0/2577251008 shutdown_connections 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.045+0000 7f7bacd0e640 1 -- 172.21.15.155:0/2577251008 wait complete. 2024-03-29T11:34:35.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 10 monclient: build_initial_monmap 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 10 monclient: monmap: 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 1 librados: starting msgr at 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 1 librados: starting objecter 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 1 Processor -- start 2024-03-29T11:34:35.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 1 -- start start 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 1 librados: setting wanted keys 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 1 librados: calling monclient init 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 10 monclient: init 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 10 monclient(hunting): picked mon.noname-b con 0x7f7ba80a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.046+0000 7f7bacd0e640 10 monclient(hunting): picked mon.noname-a con 0x7f7ba813f650 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7bacd0e640 10 monclient(hunting): picked mon.noname-c con 0x7f7ba81457f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7bacd0e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba81387a0 con 0x7f7ba813f650 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7bacd0e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba813fda0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7bacd0e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba80acd20 con 0x7f7ba81457f0 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7bacd0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7bacd0e640 10 monclient(hunting): authenticate will time out at 857.172922s 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7ba5d9b640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7ba813f650 0x7f7ba8137980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36288/0 (socket says 172.21.15.155:36288) 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7ba5d9b640 1 -- 172.21.15.155:0/973422154 learned_addr learned my addr 172.21.15.155:0/973422154 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:35.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 client.?.objecter ms_handle_connect 0x7f7ba813f650 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 client.?.objecter ms_handle_connect 0x7f7ba81457f0 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 client.?.objecter ms_handle_connect 0x7f7ba80a9d70 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372797575 0 0) 0x7f7ba81387a0 con 0x7f7ba813f650 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b88004eb0 con 0x7f7ba813f650 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143642715 0 0) 0x7f7ba80acd20 con 0x7f7ba81457f0 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba81387a0 con 0x7f7ba81457f0 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943447655 0 0) 0x7f7ba813fda0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba80acd20 con 0x7f7ba80a9d70 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2788064608 0 0) 0x7f7b88004eb0 con 0x7f7ba813f650 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ba813fda0 con 0x7f7ba813f650 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1848451398 0 0) 0x7f7ba80acd20 con 0x7f7ba80a9d70 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b88004eb0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7b9c001a10 con 0x7f7ba813f650 2024-03-29T11:34:35.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 680798432 0 0) 0x7f7ba81387a0 con 0x7f7ba81457f0 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ba80acd20 con 0x7f7ba81457f0 2024-03-29T11:34:35.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7b98001670 con 0x7f7ba80a9d70 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:35.059 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7ba0001e60 con 0x7f7ba81457f0 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:35.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2018184859 0 0) 0x7f7b88004eb0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 >> v1:172.21.15.155:6790/0 conn(0x7f7ba81457f0 legacy=0x7f7ba8138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 >> v1:172.21.15.155:6789/0 conn(0x7f7ba813f650 legacy=0x7f7ba8137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 monclient: found mon.b 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba80ad000 con 0x7f7ba80a9d70 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7bacd0e640 5 monclient: authenticate success, global_id 4189 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7bacd0e640 10 monclient: _renew_subs 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7bacd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:35.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.047+0000 7f7bacd0e640 1 -- 172.21.15.155:0/973422154 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba8142b10 con 0x7f7ba80a9d70 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7bacd0e640 10 client.4189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7bacd0e640 10 monclient: _renew_subs 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7bacd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7bacd0e640 1 -- 172.21.15.155:0/973422154 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba80ac1c0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b980017d0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7bacd0e640 1 librados: init done 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7b980047e0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 10 monclient: dump: 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.062 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7ba4d99640 4 set_mon_vals no callback set 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7b98004cd0 con 0x7f7ba80a9d70 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f7b98005f00 con 0x7f7ba80a9d70 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 10 client.4189.objecter ms_dispatch 0x7f7ba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 3 client.4189.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 3 client.4189.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7b96ffd640 20 client.4189.objecter dump_active .. 0 homeless 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7bacd0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7bacd0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:35.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7bacd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:35.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.048+0000 7f7bacd0e640 1 -- 172.21.15.155:0/973422154 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba8075f10 con 0x7f7ba80a9d70 2024-03-29T11:34:35.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.051+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7b98006980 con 0x7f7ba80a9d70 2024-03-29T11:34:35.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.051+0000 7f7b96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:35.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.051+0000 7f7b96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:35.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.113+0000 7f7bacd0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:35.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.113+0000 7f7bacd0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:35.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.113+0000 7f7bacd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:35.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.113+0000 7f7bacd0e640 1 -- 172.21.15.155:0/973422154 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ba80b3680 con 0x7f7ba80a9d70 2024-03-29T11:34:35.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.113+0000 7f7b96ffd640 1 -- 172.21.15.155:0/973422154 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f7b98006270 con 0x7f7ba80a9d70 2024-03-29T11:34:35.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.113+0000 7f7b96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:35.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.113+0000 7f7b96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:35.111 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:34:35.111 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:35.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.114+0000 7f7bacd0e640 10 librados: watch_flush enter 2024-03-29T11:34:35.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.114+0000 7f7bacd0e640 10 librados: watch_flush exit 2024-03-29T11:34:35.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.114+0000 7f7bacd0e640 20 client.4189.objecter shutdown clearing up homeless session... 2024-03-29T11:34:35.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.114+0000 7f7bacd0e640 10 client.4189.objecter successfully canceled tick 2024-03-29T11:34:35.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.114+0000 7f7bacd0e640 10 monclient: shutdown 2024-03-29T11:34:35.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.114+0000 7f7bacd0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:35.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.114+0000 7f7bacd0e640 1 -- 172.21.15.155:0/973422154 >> v1:172.21.15.183:6789/0 conn(0x7f7ba80a9d70 legacy=0x7f7ba81d72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.114+0000 7f7bacd0e640 1 -- 172.21.15.155:0/973422154 shutdown_connections 2024-03-29T11:34:35.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.114+0000 7f7bacd0e640 1 -- 172.21.15.155:0/973422154 >> 172.21.15.155:0/973422154 conn(0x7f7ba8066a00 msgr2=0x7f7ba813c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:35.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.114+0000 7f7bacd0e640 1 -- 172.21.15.155:0/973422154 shutdown_connections 2024-03-29T11:34:35.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.115+0000 7f7bacd0e640 1 -- 172.21.15.155:0/973422154 wait complete. 2024-03-29T11:34:35.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:35.115+0000 7f7bacd0e640 1 librados: shutdown 2024-03-29T11:34:35.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:35.534 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:35.537+0000 7fa948e10040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:34:35.539 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:35.542+0000 7f20ed843040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:34:35.810 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:35.814+0000 7fa948e10040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:34:35.814 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:35.817+0000 7f20ed843040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:34:35.861 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:35.865+0000 7fa948e10040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:34:35.865 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:35.867+0000 7f20ed843040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:34:35.979 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:35.983+0000 7fa948e10040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:34:35.983 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T11:34:35.986+0000 7f20ed843040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:34:37.000 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:37.003+0000 7fa8e967d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:34:37.000 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T11:34:37.003+0000 7fa8e967d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:34:38.007 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-29T11:34:38.009+0000 7f05df4aa640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:34:38.015 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-03-29T11:34:38.018+0000 7f986d619640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:34:38.036 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-29T11:34:38.038+0000 7f65c9c0d640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:34:38.058 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-03-29T11:34:38.062+0000 7fc8ea906640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:34:38.060 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-03-29T11:34:38.063+0000 7fa02c21c640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:34:38.067 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-03-29T11:34:38.070+0000 7f4e90706640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:34:38.076 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-29T11:34:38.078+0000 7fb5bcd91640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:34:38.077 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-29T11:34:38.080+0000 7f42c4cb1640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:34:41.930 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:42.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.082+0000 7f2e56a20640 10 monclient: get_monmap_and_config 2024-03-29T11:34:42.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.082+0000 7f2e56a20640 10 monclient: build_initial_monmap 2024-03-29T11:34:42.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.082+0000 7f2e56a20640 10 monclient: monmap: 2024-03-29T11:34:42.079 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:34:42.079 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:42.080 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:34:42.082872+0000 2024-03-29T11:34:42.080 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:34:42.082872+0000 2024-03-29T11:34:42.080 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:42.080 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:42.080 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:42.080 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:42.080 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:42.080 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:42.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.083+0000 7f2e56a20640 1 Processor -- start 2024-03-29T11:34:42.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.083+0000 7f2e56a20640 1 -- start start 2024-03-29T11:34:42.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.083+0000 7f2e56a20640 10 monclient: init 2024-03-29T11:34:42.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e56a20640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:42.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e56a20640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:42.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e56a20640 10 monclient(hunting): picked mon.noname-a con 0x7f2e480389e0 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:42.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e56a20640 10 monclient(hunting): picked mon.noname-b con 0x7f2e48038dc0 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e56a20640 10 monclient(hunting): picked mon.noname-c con 0x7f2e48030660 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e56a20640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e480c54c0 con 0x7f2e480389e0 2024-03-29T11:34:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e56a20640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e48036430 con 0x7f2e48030660 2024-03-29T11:34:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e56a20640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e48039840 con 0x7f2e48038dc0 2024-03-29T11:34:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e56a20640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e56a20640 10 monclient(hunting): authenticate will time out at 864.210585s 2024-03-29T11:34:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e5521d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f2e48038dc0 0x7f2e480cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:43924/0 (socket says 172.21.15.155:43924) 2024-03-29T11:34:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e5621f640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f2e48030660 0x7f2e480d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:39206/0 (socket says 172.21.15.155:39206) 2024-03-29T11:34:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.084+0000 7f2e5521d640 1 -- 172.21.15.155:0/2195250291 learned_addr learned my addr 172.21.15.155:0/2195250291 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:42.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.085+0000 7f2e54a1c640 1 -- 172.21.15.155:0/2195250291 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537038007 0 0) 0x7f2e48036430 con 0x7f2e48030660 2024-03-29T11:34:42.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.085+0000 7f2e54a1c640 10 monclient(hunting): _negotiate 2024-03-29T11:34:42.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.085+0000 7f2e54a1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:42.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.085+0000 7f2e54a1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:42.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.085+0000 7f2e54a1c640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:34:42.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.085+0000 7f2e54a1c640 1 -- 172.21.15.155:0/2195250291 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e3c003520 con 0x7f2e48030660 2024-03-29T11:34:42.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 1 -- 172.21.15.155:0/2195250291 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2894400142 0 0) 0x7f2e3c003520 con 0x7f2e48030660 2024-03-29T11:34:42.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 1 -- 172.21.15.155:0/2195250291 >> v1:172.21.15.155:6790/0 conn(0x7f2e48038dc0 legacy=0x7f2e480cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 1 -- 172.21.15.155:0/2195250291 >> v1:172.21.15.155:6789/0 conn(0x7f2e480389e0 legacy=0x7f2e48033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 10 monclient: _finish_hunting 0 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 1 monclient: found mon.noname-c 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 1 -- 172.21.15.155:0/2195250291 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e48035690 con 0x7f2e48030660 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 10 monclient: _finish_auth 0 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 10 monclient: _check_auth_tickets 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e56a20640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e56a20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 1 -- 172.21.15.155:0/2195250291 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2e40001690 con 0x7f2e48030660 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 10 monclient: dump: 2024-03-29T11:34:42.085 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:42.085 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:42.085 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:42.085 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:42.085 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.085 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:42.085 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:42.085 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:42.085 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:42.085 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:42.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e56a20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:42.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 1 -- 172.21.15.155:0/2195250291 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e400018b0 con 0x7f2e48030660 2024-03-29T11:34:42.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e56a20640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e56a20640 4 set_mon_vals no callback set 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 1 -- 172.21.15.155:0/2195250291 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2e40001690 con 0x7f2e48030660 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.086+0000 7f2e54a1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.087+0000 7f2e54a1c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.087+0000 7f2e54a1c640 10 monclient: dump: 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:42.092 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.087+0000 7f2e56a20640 10 monclient: shutdown 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.087+0000 7f2e56a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.087+0000 7f2e56a20640 1 -- 172.21.15.155:0/2195250291 >> v1:172.21.15.183:6789/0 conn(0x7f2e48030660 legacy=0x7f2e480d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.087+0000 7f2e56a20640 1 -- 172.21.15.155:0/2195250291 shutdown_connections 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.087+0000 7f2e56a20640 1 -- 172.21.15.155:0/2195250291 >> 172.21.15.155:0/2195250291 conn(0x7f2e48005c00 msgr2=0x7f2e48006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.088+0000 7f2e56a20640 1 -- 172.21.15.155:0/2195250291 shutdown_connections 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.088+0000 7f2e56a20640 1 -- 172.21.15.155:0/2195250291 wait complete. 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.088+0000 7f2e56a20640 10 monclient: build_initial_monmap 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.088+0000 7f2e56a20640 10 monclient: monmap: 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:34:42.093 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.089+0000 7f2e56a20640 1 librados: starting msgr at 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.089+0000 7f2e56a20640 1 librados: starting objecter 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.089+0000 7f2e56a20640 1 Processor -- start 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.089+0000 7f2e56a20640 1 -- start start 2024-03-29T11:34:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.089+0000 7f2e56a20640 1 librados: setting wanted keys 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.089+0000 7f2e56a20640 1 librados: calling monclient init 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.089+0000 7f2e56a20640 10 monclient: init 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e56a20640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e56a20640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e56a20640 10 monclient(hunting): picked mon.noname-a con 0x7f2e48030660 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e56a20640 10 monclient(hunting): picked mon.noname-c con 0x7f2e480389e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e56a20640 10 monclient(hunting): picked mon.noname-b con 0x7f2e481609b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e56a20640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4802ea70 con 0x7f2e48030660 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e56a20640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e48030070 con 0x7f2e481609b0 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e56a20640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4815fda0 con 0x7f2e480389e0 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e56a20640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e56a20640 10 monclient(hunting): authenticate will time out at 864.216527s 2024-03-29T11:34:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e5621f640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f2e481609b0 0x7f2e4802f960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:39332/0 (socket says 172.21.15.155:39332) 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e55a1e640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f2e48030660 0x7f2e481602a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:39446/0 (socket says 172.21.15.155:39446) 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.090+0000 7f2e5621f640 1 -- 172.21.15.155:0/349995791 learned_addr learned my addr 172.21.15.155:0/349995791 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 10 client.?.objecter ms_handle_connect 0x7f2e481609b0 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 10 client.?.objecter ms_handle_connect 0x7f2e48030660 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 10 client.?.objecter ms_handle_connect 0x7f2e480389e0 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939944949 0 0) 0x7f2e48030070 con 0x7f2e481609b0 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.091+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e34004eb0 con 0x7f2e481609b0 2024-03-29T11:34:42.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 666924930 0 0) 0x7f2e34004eb0 con 0x7f2e481609b0 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e48030070 con 0x7f2e481609b0 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2e40001690 con 0x7f2e481609b0 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 10 monclient(hunting): dump: 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:42.097 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720039757 0 0) 0x7f2e4802ea70 con 0x7f2e48030660 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.092+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e34004eb0 con 0x7f2e48030660 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 803005821 0 0) 0x7f2e48030070 con 0x7f2e481609b0 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 >> v1:172.21.15.155:6790/0 conn(0x7f2e480389e0 legacy=0x7f2e4802f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 >> v1:172.21.15.155:6789/0 conn(0x7f2e48030660 legacy=0x7f2e481602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e467fc640 1 monclient: found mon.b 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e467fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e480c5650 con 0x7f2e481609b0 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e467fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e56a20640 5 monclient: authenticate success, global_id 4234 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e56a20640 10 monclient: _renew_subs 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e56a20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e56a20640 1 -- 172.21.15.155:0/349995791 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e480c85a0 con 0x7f2e481609b0 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e56a20640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e56a20640 10 monclient: _renew_subs 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e56a20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e56a20640 1 -- 172.21.15.155:0/349995791 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4802d5a0 con 0x7f2e481609b0 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.093+0000 7f2e56a20640 1 librados: init done 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.094+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e400011e0 con 0x7f2e481609b0 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.094+0000 7f2e467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.094+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2e40004ce0 con 0x7f2e481609b0 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.094+0000 7f2e467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.094+0000 7f2e467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.094+0000 7f2e467fc640 10 monclient: dump: 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:42.100 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.101 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:42.101 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:42.101 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:42.101 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:42.101 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:42.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.094+0000 7f2e477fe640 4 set_mon_vals no callback set 2024-03-29T11:34:42.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.094+0000 7f2e56a20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.095+0000 7f2e56a20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.095+0000 7f2e56a20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.095+0000 7f2e56a20640 1 -- 172.21.15.155:0/349995791 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e38005080 con 0x7f2e481609b0 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.096+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f2e4001acb0 con 0x7f2e481609b0 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.097+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f2e40084be0 con 0x7f2e481609b0 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.097+0000 7f2e467fc640 10 client.4234.objecter ms_dispatch 0x7f2e50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.097+0000 7f2e467fc640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.097+0000 7f2e467fc640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.097+0000 7f2e467fc640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.097+0000 7f2e55a1e640 10 monclient: get_auth_request con 0x7f2e3406dce0 auth_method 0 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.097+0000 7f2e467fc640 10 client.4234.objecter ms_handle_connect 0x7f2e3406dce0 2024-03-29T11:34:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.103+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e400502f0 con 0x7f2e481609b0 2024-03-29T11:34:42.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.103+0000 7f2e467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:42.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.103+0000 7f2e467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:42.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.489+0000 7f2e56a20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:42.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.489+0000 7f2e56a20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:42.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.489+0000 7f2e56a20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:42.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.489+0000 7f2e56a20640 1 -- 172.21.15.155:0/349995791 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e38005330 con 0x7f2e481609b0 2024-03-29T11:34:42.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.491+0000 7f2e467fc640 1 -- 172.21.15.155:0/349995791 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f2e400523f0 con 0x7f2e481609b0 2024-03-29T11:34:42.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.491+0000 7f2e467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:42.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.491+0000 7f2e467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:42.489 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:34:42.489 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:42.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.494+0000 7f2e56a20640 10 librados: watch_flush enter 2024-03-29T11:34:42.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.494+0000 7f2e56a20640 10 librados: watch_flush exit 2024-03-29T11:34:42.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.494+0000 7f2e56a20640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-29T11:34:42.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.494+0000 7f2e56a20640 10 client.4234.objecter successfully canceled tick 2024-03-29T11:34:42.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.494+0000 7f2e56a20640 1 -- 172.21.15.155:0/349995791 >> v1:172.21.15.155:6816/1760814708 conn(0x7f2e3406dce0 legacy=0x7f2e340701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.494+0000 7f2e56a20640 10 monclient: shutdown 2024-03-29T11:34:42.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.494+0000 7f2e56a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:42.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.495+0000 7f2e56a20640 1 -- 172.21.15.155:0/349995791 >> v1:172.21.15.183:6789/0 conn(0x7f2e481609b0 legacy=0x7f2e4802f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.495+0000 7f2e56a20640 1 -- 172.21.15.155:0/349995791 shutdown_connections 2024-03-29T11:34:42.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.495+0000 7f2e56a20640 1 -- 172.21.15.155:0/349995791 >> 172.21.15.155:0/349995791 conn(0x7f2e48005c00 msgr2=0x7f2e48034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:42.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.495+0000 7f2e56a20640 1 -- 172.21.15.155:0/349995791 shutdown_connections 2024-03-29T11:34:42.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.496+0000 7f2e56a20640 1 -- 172.21.15.155:0/349995791 wait complete. 2024-03-29T11:34:42.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:42.496+0000 7f2e56a20640 1 librados: shutdown 2024-03-29T11:34:42.507 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:49.315 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:49.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.480+0000 7f990494b640 10 monclient: get_monmap_and_config 2024-03-29T11:34:49.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.480+0000 7f990494b640 10 monclient: build_initial_monmap 2024-03-29T11:34:49.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.480+0000 7f990494b640 10 monclient: monmap: 2024-03-29T11:34:49.478 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:34:49.478 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:49.478 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:34:49.480939+0000 2024-03-29T11:34:49.478 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:34:49.480939+0000 2024-03-29T11:34:49.478 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:49.478 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:49.478 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:49.478 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:49.479 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:49.479 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:49.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.481+0000 7f990494b640 1 Processor -- start 2024-03-29T11:34:49.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.481+0000 7f990494b640 1 -- start start 2024-03-29T11:34:49.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.481+0000 7f990494b640 10 monclient: init 2024-03-29T11:34:49.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f990494b640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f990494b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f990494b640 10 monclient(hunting): picked mon.noname-b con 0x7f99000b0770 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f990494b640 10 monclient(hunting): picked mon.noname-a con 0x7f990014a700 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f990494b640 10 monclient(hunting): picked mon.noname-c con 0x7f99000ac5a0 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f990494b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99000aba70 con 0x7f990014a700 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f990494b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99000aecb0 con 0x7f99000ac5a0 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f990494b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99000af150 con 0x7f99000b0770 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f990494b640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f990494b640 10 monclient(hunting): authenticate will time out at 871.608458s 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f98fed9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f99000ac5a0 0x7f9900151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:56480/0 (socket says 172.21.15.155:56480) 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.482+0000 7f98fed9d640 1 -- 172.21.15.155:0/2977230424 learned_addr learned my addr 172.21.15.155:0/2977230424 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.483+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152016146 0 0) 0x7f99000aecb0 con 0x7f99000ac5a0 2024-03-29T11:34:49.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.483+0000 7f98fd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.483+0000 7f98fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.483+0000 7f98fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.483+0000 7f98fd59a640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:34:49.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.483+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ec003520 con 0x7f99000ac5a0 2024-03-29T11:34:49.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.483+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618827287 0 0) 0x7f99000aba70 con 0x7f990014a700 2024-03-29T11:34:49.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.483+0000 7f98fd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.483+0000 7f98fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.483+0000 7f98fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.483+0000 7f98fd59a640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:34:49.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ec003840 con 0x7f990014a700 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814724446 0 0) 0x7f99000af150 con 0x7f99000b0770 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ec003b80 con 0x7f99000b0770 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2474943422 0 0) 0x7f98ec003840 con 0x7f990014a700 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 >> v1:172.21.15.155:6790/0 conn(0x7f99000b0770 legacy=0x7f9900149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 >> v1:172.21.15.183:6789/0 conn(0x7f99000ac5a0 legacy=0x7f9900151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 1 monclient: found mon.noname-a 2024-03-29T11:34:49.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99000ac9d0 con 0x7f990014a700 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 10 monclient: _finish_auth 0 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f990494b640 5 monclient: authenticate success, global_id 4242 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f990494b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f98e80015f0 con 0x7f990014a700 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f98fd59a640 10 monclient: dump: 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:49.485 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.484+0000 7f990494b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.485+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98e80017d0 con 0x7f990014a700 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.485+0000 7f98fd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.485+0000 7f990494b640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.485+0000 7f990494b640 4 set_mon_vals no callback set 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.485+0000 7f98fd59a640 1 -- 172.21.15.155:0/2977230424 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f98e80015f0 con 0x7f990014a700 2024-03-29T11:34:49.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.485+0000 7f98fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.485+0000 7f98fd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:49.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.485+0000 7f98fd59a640 10 monclient: dump: 2024-03-29T11:34:49.487 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:49.487 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:49.487 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:49.487 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:49.487 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.487 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:49.487 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:49.487 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:49.487 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.485+0000 7f990494b640 10 monclient: shutdown 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.485+0000 7f990494b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.485+0000 7f990494b640 1 -- 172.21.15.155:0/2977230424 >> v1:172.21.15.155:6789/0 conn(0x7f990014a700 legacy=0x7f99000abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.486+0000 7f990494b640 1 -- 172.21.15.155:0/2977230424 shutdown_connections 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.486+0000 7f990494b640 1 -- 172.21.15.155:0/2977230424 >> 172.21.15.155:0/2977230424 conn(0x7f990006b660 msgr2=0x7f990006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.486+0000 7f990494b640 1 -- 172.21.15.155:0/2977230424 shutdown_connections 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.486+0000 7f990494b640 1 -- 172.21.15.155:0/2977230424 wait complete. 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.487+0000 7f990494b640 10 monclient: build_initial_monmap 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.487+0000 7f990494b640 10 monclient: monmap: 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:34:49.488 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:34:49.489 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:49.489 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:49.489 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:49.489 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:49.489 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:49.489 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:49.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.487+0000 7f990494b640 1 librados: starting msgr at 2024-03-29T11:34:49.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.487+0000 7f990494b640 1 librados: starting objecter 2024-03-29T11:34:49.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.487+0000 7f990494b640 1 Processor -- start 2024-03-29T11:34:49.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.487+0000 7f990494b640 1 -- start start 2024-03-29T11:34:49.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.487+0000 7f990494b640 1 librados: setting wanted keys 2024-03-29T11:34:49.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.487+0000 7f990494b640 1 librados: calling monclient init 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.487+0000 7f990494b640 10 monclient: init 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f990494b640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f990494b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f990494b640 10 monclient(hunting): picked mon.noname-c con 0x7f99000ac5a0 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f990494b640 10 monclient(hunting): picked mon.noname-b con 0x7f99000b0770 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f990494b640 10 monclient(hunting): picked mon.noname-a con 0x7f990014a700 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f990494b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990012ab20 con 0x7f990014a700 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f990494b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990012ae00 con 0x7f99000b0770 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f990494b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990012b0e0 con 0x7f99000ac5a0 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f990494b640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f990494b640 10 monclient(hunting): authenticate will time out at 871.614686s 2024-03-29T11:34:49.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f98fe59c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f99000ac5a0 0x7f9900147cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:35396/0 (socket says 172.21.15.155:35396) 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.488+0000 7f98fe59c640 1 -- 172.21.15.155:0/2570325063 learned_addr learned my addr 172.21.15.155:0/2570325063 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 10 client.?.objecter ms_handle_connect 0x7f99000ac5a0 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 10 client.?.objecter ms_handle_connect 0x7f990014a700 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 10 client.?.objecter ms_handle_connect 0x7f99000b0770 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465206111 0 0) 0x7f990012ab20 con 0x7f990014a700 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:34:49.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.489+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e0004eb0 con 0x7f990014a700 2024-03-29T11:34:49.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194739564 0 0) 0x7f990012b0e0 con 0x7f99000ac5a0 2024-03-29T11:34:49.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:34:49.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f990012ab20 con 0x7f99000ac5a0 2024-03-29T11:34:49.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017759111 0 0) 0x7f990012ae00 con 0x7f99000b0770 2024-03-29T11:34:49.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:34:49.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f990012b0e0 con 0x7f99000b0770 2024-03-29T11:34:49.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4153273885 0 0) 0x7f98e0004eb0 con 0x7f990014a700 2024-03-29T11:34:49.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f990012ae00 con 0x7f990014a700 2024-03-29T11:34:49.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f98f80018b0 con 0x7f990014a700 2024-03-29T11:34:49.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:49.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2946849228 0 0) 0x7f990012ae00 con 0x7f990014a700 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 >> v1:172.21.15.155:6790/0 conn(0x7f99000ac5a0 legacy=0x7f9900147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 >> v1:172.21.15.183:6789/0 conn(0x7f99000b0770 legacy=0x7f99001db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 monclient: found mon.a 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990012a260 con 0x7f990014a700 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f98f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f990494b640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f990494b640 10 monclient: _renew_subs 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f990494b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f990494b640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9900129da0 con 0x7f990014a700 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f990494b640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f990494b640 10 monclient: _renew_subs 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f990494b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:49.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f990494b640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99001291b0 con 0x7f990014a700 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.490+0000 7f990494b640 1 librados: init done 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.491+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98f8001cc0 con 0x7f990014a700 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.491+0000 7f98f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.491+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f98f8004980 con 0x7f990014a700 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.491+0000 7f98f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.491+0000 7f98f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.491+0000 7f98f6ffd640 10 monclient: dump: 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:49.505 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.491+0000 7f98fcd99640 4 set_mon_vals no callback set 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.492+0000 7f990494b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.492+0000 7f990494b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.492+0000 7f990494b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.492+0000 7f990494b640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e4005080 con 0x7f990014a700 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.493+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f98f801a950 con 0x7f990014a700 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.493+0000 7f98fed9d640 1 -- 172.21.15.155:0/2570325063 >> v1:172.21.15.155:6789/0 conn(0x7f990014a700 legacy=0x7f99001dbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.493+0000 7f98fed9d640 1 -- 172.21.15.155:0/2570325063 >> v1:172.21.15.155:6789/0 conn(0x7f990014a700 legacy=0x7f99001dbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.493+0000 7f98fed9d640 1 --1- 172.21.15.155:0/2570325063 >> v1:172.21.15.155:6789/0 conn(0x7f990014a700 0x7f99001dbf50 :-1 s=OPENED pgs=52145 cs=1 l=1).handle_message read tag failed 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.493+0000 7f98fed9d640 1 --1- 172.21.15.155:0/2570325063 >> v1:172.21.15.155:6789/0 conn(0x7f990014a700 0x7f99001dbf50 :-1 s=OPENED pgs=52145 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.155:6789/0 2024-03-29T11:34:49.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 >> v1:172.21.15.155:6789/0 conn(0x7f990014a700 legacy=0x7f99001dbf50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 10 monclient(hunting): picked mon.b con 0x7f99000b0770 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 10 monclient(hunting): picked mon.c con 0x7f99000ac5a0 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 10 monclient(hunting): picked mon.a con 0x7f98e0078ea0 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f990012ae00 con 0x7f98e0078ea0 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f98e0004eb0 con 0x7f99000b0770 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f98e007eff0 con 0x7f99000ac5a0 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98fe59c640 10 monclient(hunting): get_auth_request con 0x7f98e006de10 auth_method 0 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.494+0000 7f98f6ffd640 10 client.4245.objecter ms_handle_connect 0x7f98e006de10 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 client.4245.objecter ms_handle_connect 0x7f99000b0770 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 client.4245.objecter resend_mon_ops 2024-03-29T11:34:49.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 client.4245.objecter ms_handle_connect 0x7f98e0078ea0 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 client.4245.objecter resend_mon_ops 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 client.4245.objecter ms_handle_connect 0x7f99000ac5a0 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 client.4245.objecter resend_mon_ops 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555105971 0 0) 0x7f98e0004eb0 con 0x7f99000b0770 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.183:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f98e007cf90 con 0x7f99000b0770 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69011094 0 0) 0x7f98e007eff0 con 0x7f99000ac5a0 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.155:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f98e0004eb0 con 0x7f99000ac5a0 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2470240731 0 0) 0x7f98e007cf90 con 0x7f99000b0770 2024-03-29T11:34:49.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 >> v1:172.21.15.155:6790/0 conn(0x7f99000ac5a0 legacy=0x7f98e0077420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 >> v1:172.21.15.155:6789/0 conn(0x7f98e0078ea0 legacy=0x7f98e007b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 1 monclient: found mon.b 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f98e00801b0 con 0x7f99000b0770 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e0004eb0 con 0x7f99000b0770 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f98e8001190 con 0x7f99000b0770 2024-03-29T11:34:49.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.495+0000 7f98f6ffd640 10 monclient: dump: 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.496+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98e8001bf0 con 0x7f99000b0770 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.496+0000 7f98f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:49.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.496+0000 7f98f7fff640 4 set_mon_vals no callback set 2024-03-29T11:34:49.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.500+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f98e80012f0 con 0x7f99000b0770 2024-03-29T11:34:49.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.501+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.1 v1:172.21.15.183:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f98e8082df0 con 0x7f99000b0770 2024-03-29T11:34:49.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.501+0000 7f98f6ffd640 10 client.4245.objecter ms_dispatch 0x7f9900000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:49.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.501+0000 7f98f6ffd640 3 client.4245.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:49.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.501+0000 7f98f6ffd640 3 client.4245.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:49.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.501+0000 7f98f6ffd640 20 client.4245.objecter dump_active .. 0 homeless 2024-03-29T11:34:49.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.504+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98e8052bc0 con 0x7f99000b0770 2024-03-29T11:34:49.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.504+0000 7f98f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:49.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.504+0000 7f98f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:49.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.844+0000 7f990494b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:49.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.844+0000 7f990494b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:49.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.844+0000 7f990494b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:49.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.844+0000 7f990494b640 1 -- 172.21.15.155:0/2570325063 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98e4005330 con 0x7f99000b0770 2024-03-29T11:34:49.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.845+0000 7f98f6ffd640 1 -- 172.21.15.155:0/2570325063 <== mon.1 v1:172.21.15.183:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f98e80524c0 con 0x7f99000b0770 2024-03-29T11:34:49.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.845+0000 7f98f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:49.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.845+0000 7f98f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:49.842 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:34:49.843 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.848+0000 7f990494b640 10 librados: watch_flush enter 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.848+0000 7f990494b640 10 librados: watch_flush exit 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.848+0000 7f990494b640 20 client.4245.objecter shutdown clearing up homeless session... 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.848+0000 7f990494b640 10 client.4245.objecter successfully canceled tick 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.848+0000 7f990494b640 1 -- 172.21.15.155:0/2570325063 >> v1:172.21.15.155:6816/1760814708 conn(0x7f98e006de10 legacy=0x7f98e00702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.848+0000 7f990494b640 10 monclient: shutdown 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.848+0000 7f990494b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.848+0000 7f990494b640 1 -- 172.21.15.155:0/2570325063 >> v1:172.21.15.183:6789/0 conn(0x7f99000b0770 legacy=0x7f98e0073af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.848+0000 7f990494b640 1 -- 172.21.15.155:0/2570325063 shutdown_connections 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.848+0000 7f990494b640 1 -- 172.21.15.155:0/2570325063 >> 172.21.15.155:0/2570325063 conn(0x7f990006b660 msgr2=0x7f990006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.849+0000 7f990494b640 1 -- 172.21.15.155:0/2570325063 shutdown_connections 2024-03-29T11:34:49.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.849+0000 7f990494b640 1 -- 172.21.15.155:0/2570325063 wait complete. 2024-03-29T11:34:49.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:49.849+0000 7f990494b640 1 librados: shutdown 2024-03-29T11:34:49.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:56.667 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:56.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.793+0000 7fb2d9e17640 10 monclient: get_monmap_and_config 2024-03-29T11:34:56.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.793+0000 7fb2d9e17640 10 monclient: build_initial_monmap 2024-03-29T11:34:56.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.793+0000 7fb2d9e17640 10 monclient: monmap: 2024-03-29T11:34:56.791 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:34:56.792 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:56.792 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:34:56.794555+0000 2024-03-29T11:34:56.792 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:34:56.794555+0000 2024-03-29T11:34:56.792 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:56.792 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:56.792 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:56.792 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:34:56.792 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:34:56.792 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:56.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.794+0000 7fb2d9e17640 1 Processor -- start 2024-03-29T11:34:56.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.794+0000 7fb2d9e17640 1 -- start start 2024-03-29T11:34:56.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.794+0000 7fb2d9e17640 10 monclient: init 2024-03-29T11:34:56.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d9e17640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:56.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d9e17640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:56.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d9e17640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d406ade0 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:56.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d9e17640 10 monclient(hunting): picked mon.noname-b con 0x7fb2d40afa30 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:56.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d9e17640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d41433e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:56.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d9e17640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4072670 con 0x7fb2d406ade0 2024-03-29T11:34:56.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d9e17640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d413da60 con 0x7fb2d41433e0 2024-03-29T11:34:56.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d9e17640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4143fa0 con 0x7fb2d40afa30 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d9e17640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d9e17640 10 monclient(hunting): authenticate will time out at 878.922026s 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d2ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fb2d40afa30 0x7fb2d413d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:56950/0 (socket says 172.21.15.155:56950) 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d2ffd640 1 -- 172.21.15.155:0/3390769207 learned_addr learned my addr 172.21.15.155:0/3390769207 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 1 -- 172.21.15.155:0/3390769207 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038812593 0 0) 0x7fb2d413da60 con 0x7fb2d41433e0 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 1 -- 172.21.15.155:0/3390769207 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2bc003520 con 0x7fb2d41433e0 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 1 -- 172.21.15.155:0/3390769207 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122049276 0 0) 0x7fb2d4143fa0 con 0x7fb2d40afa30 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:56.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:34:56.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.795+0000 7fb2d27fc640 1 -- 172.21.15.155:0/3390769207 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d413da60 con 0x7fb2d40afa30 2024-03-29T11:34:56.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.796+0000 7fb2d27fc640 1 -- 172.21.15.155:0/3390769207 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1716920095 0 0) 0x7fb2bc003520 con 0x7fb2d41433e0 2024-03-29T11:34:56.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.796+0000 7fb2d27fc640 1 -- 172.21.15.155:0/3390769207 >> v1:172.21.15.155:6790/0 conn(0x7fb2d40afa30 legacy=0x7fb2d413d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:56.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.796+0000 7fb2d27fc640 1 -- 172.21.15.155:0/3390769207 >> v1:172.21.15.155:6789/0 conn(0x7fb2d406ade0 legacy=0x7fb2d4071d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:56.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:56.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 1 monclient: found mon.noname-c 2024-03-29T11:34:56.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 1 -- 172.21.15.155:0/3390769207 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d414f730 con 0x7fb2d41433e0 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 1 -- 172.21.15.155:0/3390769207 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb2cc001a10 con 0x7fb2d41433e0 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d27fc640 10 monclient: dump: 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:56.798 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d9e17640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.798+0000 7fb2d9e17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.799+0000 7fb2d27fc640 1 -- 172.21.15.155:0/3390769207 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2cc001b70 con 0x7fb2d41433e0 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.799+0000 7fb2d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.799+0000 7fb2d27fc640 1 -- 172.21.15.155:0/3390769207 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb2cc001f00 con 0x7fb2d41433e0 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.799+0000 7fb2d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.799+0000 7fb2d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.799+0000 7fb2d27fc640 10 monclient: dump: 2024-03-29T11:34:56.799 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:56.800 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:56.800 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:56.800 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:56.800 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:56.800 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:56.800 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:56.800 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:56.800 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:56.800 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:56.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.799+0000 7fb2d9e17640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:56.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.799+0000 7fb2d9e17640 4 set_mon_vals no callback set 2024-03-29T11:34:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.800+0000 7fb2d9e17640 10 monclient: shutdown 2024-03-29T11:34:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.800+0000 7fb2d9e17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.800+0000 7fb2d9e17640 1 -- 172.21.15.155:0/3390769207 >> v1:172.21.15.183:6789/0 conn(0x7fb2d41433e0 legacy=0x7fb2d4143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.800+0000 7fb2d9e17640 1 -- 172.21.15.155:0/3390769207 shutdown_connections 2024-03-29T11:34:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.800+0000 7fb2d9e17640 1 -- 172.21.15.155:0/3390769207 >> 172.21.15.155:0/3390769207 conn(0x7fb2d4067470 msgr2=0x7fb2d40b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.800+0000 7fb2d9e17640 1 -- 172.21.15.155:0/3390769207 shutdown_connections 2024-03-29T11:34:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.800+0000 7fb2d9e17640 1 -- 172.21.15.155:0/3390769207 wait complete. 2024-03-29T11:34:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.801+0000 7fb2d9e17640 10 monclient: build_initial_monmap 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.801+0000 7fb2d9e17640 10 monclient: monmap: 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.801+0000 7fb2d9e17640 1 librados: starting msgr at 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.801+0000 7fb2d9e17640 1 librados: starting objecter 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.801+0000 7fb2d9e17640 1 Processor -- start 2024-03-29T11:34:56.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.801+0000 7fb2d9e17640 1 -- start start 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.801+0000 7fb2d9e17640 1 librados: setting wanted keys 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.801+0000 7fb2d9e17640 1 librados: calling monclient init 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.801+0000 7fb2d9e17640 10 monclient: init 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.802+0000 7fb2d9e17640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.802+0000 7fb2d9e17640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.802+0000 7fb2d9e17640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d406ade0 addr v1:172.21.15.155:6789/0 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.802+0000 7fb2d9e17640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d40afa30 addr v1:172.21.15.155:6790/0 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.802+0000 7fb2d9e17640 10 monclient(hunting): picked mon.noname-b con 0x7fb2d41433e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.802+0000 7fb2d9e17640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4140d10 con 0x7fb2d406ade0 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.802+0000 7fb2d9e17640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4128330 con 0x7fb2d41433e0 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2d9e17640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d413f580 con 0x7fb2d40afa30 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2d3fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb2d41433e0 0x7fb2d40717a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:59374/0 (socket says 172.21.15.155:59374) 2024-03-29T11:34:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2d3fff640 1 -- 172.21.15.155:0/1836087920 learned_addr learned my addr 172.21.15.155:0/1836087920 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2d9e17640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2d9e17640 10 monclient(hunting): authenticate will time out at 878.929984s 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 client.?.objecter ms_handle_connect 0x7fb2d41433e0 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588097400 0 0) 0x7fb2d4128330 con 0x7fb2d41433e0 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c0004eb0 con 0x7fb2d41433e0 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 client.?.objecter ms_handle_connect 0x7fb2d40afa30 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691547784 0 0) 0x7fb2d413f580 con 0x7fb2d40afa30 2024-03-29T11:34:56.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d4128330 con 0x7fb2d40afa30 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1354489229 0 0) 0x7fb2c0004eb0 con 0x7fb2d41433e0 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2d413f580 con 0x7fb2d41433e0 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb2cc001ec0 con 0x7fb2d41433e0 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): dump: 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:56.807 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.803+0000 7fb2bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.804+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 646365710 0 0) 0x7fb2d4128330 con 0x7fb2d40afa30 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.804+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2c0004eb0 con 0x7fb2d40afa30 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.804+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb2c8001930 con 0x7fb2d40afa30 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.804+0000 7fb2bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.804+0000 7fb2bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.804+0000 7fb2bbfff640 10 monclient(hunting): dump: 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.804+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4189604828 0 0) 0x7fb2d413f580 con 0x7fb2d41433e0 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.804+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 >> v1:172.21.15.155:6790/0 conn(0x7fb2d40afa30 legacy=0x7fb2d4128850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:56.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.804+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 >> v1:172.21.15.155:6789/0 conn(0x7fb2d406ade0 legacy=0x7fb2d413eb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:56.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2bbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:56.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2bbfff640 1 monclient: found mon.b 2024-03-29T11:34:56.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:56.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2bbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:56.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d413dc70 con 0x7fb2d41433e0 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2bbfff640 10 monclient: _finish_auth 0 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2bbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2d9e17640 5 monclient: authenticate success, global_id 4246 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2d9e17640 10 monclient: _renew_subs 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2d9e17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.805+0000 7fb2d9e17640 1 -- 172.21.15.155:0/1836087920 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d413f230 con 0x7fb2d41433e0 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2d9e17640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2d9e17640 10 monclient: _renew_subs 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2d9e17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2d9e17640 1 -- 172.21.15.155:0/1836087920 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d4139130 con 0x7fb2d41433e0 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2d9e17640 1 librados: init done 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2cc0020a0 con 0x7fb2d41433e0 2024-03-29T11:34:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb2cc003dc0 con 0x7fb2d41433e0 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2bbfff640 10 monclient: dump: 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:34:56.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.806+0000 7fb2d1ffb640 4 set_mon_vals no callback set 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.807+0000 7fb2d9e17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.807+0000 7fb2d9e17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.807+0000 7fb2d9e17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.807+0000 7fb2d9e17640 1 -- 172.21.15.155:0/1836087920 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b4005080 con 0x7fb2d41433e0 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.808+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fb2cc003f20 con 0x7fb2d41433e0 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.808+0000 7fb2d37fe640 10 monclient: get_auth_request con 0x7fb2c006ded0 auth_method 0 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.808+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fb2cc084e50 con 0x7fb2d41433e0 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.808+0000 7fb2bbfff640 10 client.4246.objecter ms_dispatch 0x7fb2d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.808+0000 7fb2bbfff640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.808+0000 7fb2bbfff640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.808+0000 7fb2bbfff640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-29T11:34:56.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.809+0000 7fb2bbfff640 10 client.4246.objecter ms_handle_connect 0x7fb2c006ded0 2024-03-29T11:34:56.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.816+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2cc050730 con 0x7fb2d41433e0 2024-03-29T11:34:56.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.816+0000 7fb2bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:56.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:56.816+0000 7fb2bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:57.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.101+0000 7fb2d9e17640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.101+0000 7fb2d9e17640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.101+0000 7fb2d9e17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:34:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.101+0000 7fb2d9e17640 1 -- 172.21.15.155:0/1836087920 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2b4005330 con 0x7fb2d41433e0 2024-03-29T11:34:57.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.103+0000 7fb2bbfff640 1 -- 172.21.15.155:0/1836087920 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fb2cc052660 con 0x7fb2d41433e0 2024-03-29T11:34:57.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.103+0000 7fb2bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:57.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.103+0000 7fb2bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:57.101 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:34:57.101 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:57.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.106+0000 7fb2d9e17640 10 librados: watch_flush enter 2024-03-29T11:34:57.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.106+0000 7fb2d9e17640 10 librados: watch_flush exit 2024-03-29T11:34:57.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.106+0000 7fb2d9e17640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-29T11:34:57.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.106+0000 7fb2d9e17640 10 client.4246.objecter successfully canceled tick 2024-03-29T11:34:57.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.106+0000 7fb2d9e17640 1 -- 172.21.15.155:0/1836087920 >> v1:172.21.15.155:6816/1760814708 conn(0x7fb2c006ded0 legacy=0x7fb2c0070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.106+0000 7fb2d9e17640 10 monclient: shutdown 2024-03-29T11:34:57.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.106+0000 7fb2d9e17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:57.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.106+0000 7fb2d9e17640 1 -- 172.21.15.155:0/1836087920 >> v1:172.21.15.183:6789/0 conn(0x7fb2d41433e0 legacy=0x7fb2d40717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.106+0000 7fb2d9e17640 1 -- 172.21.15.155:0/1836087920 shutdown_connections 2024-03-29T11:34:57.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.106+0000 7fb2d9e17640 1 -- 172.21.15.155:0/1836087920 >> 172.21.15.155:0/1836087920 conn(0x7fb2d4067470 msgr2=0x7fb2d413b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:57.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.107+0000 7fb2d9e17640 1 -- 172.21.15.155:0/1836087920 shutdown_connections 2024-03-29T11:34:57.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.107+0000 7fb2d9e17640 1 -- 172.21.15.155:0/1836087920 wait complete. 2024-03-29T11:34:57.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:34:57.107+0000 7fb2d9e17640 1 librados: shutdown 2024-03-29T11:34:57.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:03.926 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:04.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.080+0000 7f1612548640 10 monclient: get_monmap_and_config 2024-03-29T11:35:04.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.080+0000 7f1612548640 10 monclient: build_initial_monmap 2024-03-29T11:35:04.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.080+0000 7f1612548640 10 monclient: monmap: 2024-03-29T11:35:04.079 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:04.079 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:04.079 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:35:04.081872+0000 2024-03-29T11:35:04.079 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:35:04.081872+0000 2024-03-29T11:35:04.079 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:04.079 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:04.079 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:04.079 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:35:04.079 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:35:04.079 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:04.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.082+0000 7f1612548640 1 Processor -- start 2024-03-29T11:35:04.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.083+0000 7f1612548640 1 -- start start 2024-03-29T11:35:04.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.083+0000 7f1612548640 10 monclient: init 2024-03-29T11:35:04.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.083+0000 7f1612548640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:04.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.083+0000 7f1612548640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:04.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.083+0000 7f1612548640 10 monclient(hunting): picked mon.noname-b con 0x7f160c1457f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:04.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.083+0000 7f1612548640 10 monclient(hunting): picked mon.noname-a con 0x7f160c13f650 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:04.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.084+0000 7f1612548640 10 monclient(hunting): picked mon.noname-c con 0x7f160c0a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:04.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.084+0000 7f1612548640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c0aaef0 con 0x7f160c13f650 2024-03-29T11:35:04.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.084+0000 7f1612548640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c145bd0 con 0x7f160c0a9d70 2024-03-29T11:35:04.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.084+0000 7f1612548640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c145eb0 con 0x7f160c1457f0 2024-03-29T11:35:04.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.084+0000 7f1612548640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:04.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.084+0000 7f1612548640 10 monclient(hunting): authenticate will time out at 886.210967s 2024-03-29T11:35:04.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.090+0000 7f160bfff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f160c1457f0 0x7f160c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55721/0 (socket says 172.21.15.155:55721) 2024-03-29T11:35:04.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.090+0000 7f160bfff640 1 -- 172.21.15.155:0/699372246 learned_addr learned my addr 172.21.15.155:0/699372246 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:04.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725577618 0 0) 0x7f160c145eb0 con 0x7f160c1457f0 2024-03-29T11:35:04.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:35:04.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f4003520 con 0x7f160c1457f0 2024-03-29T11:35:04.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586899712 0 0) 0x7f160c145bd0 con 0x7f160c0a9d70 2024-03-29T11:35:04.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:35:04.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.092+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160c145eb0 con 0x7f160c0a9d70 2024-03-29T11:35:04.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798639669 0 0) 0x7f160c0aaef0 con 0x7f160c13f650 2024-03-29T11:35:04.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:35:04.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f4004a30 con 0x7f160c13f650 2024-03-29T11:35:04.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3305683195 0 0) 0x7f15f4003520 con 0x7f160c1457f0 2024-03-29T11:35:04.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 >> v1:172.21.15.183:6789/0 conn(0x7f160c0a9d70 legacy=0x7f160c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 >> v1:172.21.15.155:6789/0 conn(0x7f160c13f650 legacy=0x7f160c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 1 monclient: found mon.noname-b 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c146340 con 0x7f160c1457f0 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1600001690 con 0x7f160c1457f0 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f160affd640 10 monclient: dump: 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:04.095 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.094+0000 7f1612548640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.095+0000 7f1612548640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.095+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16000017f0 con 0x7f160c1457f0 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.095+0000 7f160affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.095+0000 7f1612548640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.095+0000 7f1612548640 4 set_mon_vals no callback set 2024-03-29T11:35:04.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.095+0000 7f160affd640 1 -- 172.21.15.155:0/699372246 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1600001c00 con 0x7f160c1457f0 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.095+0000 7f160affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.095+0000 7f160affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.095+0000 7f160affd640 10 monclient: dump: 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.096+0000 7f1612548640 10 monclient: shutdown 2024-03-29T11:35:04.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.096+0000 7f1612548640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:04.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.096+0000 7f1612548640 1 -- 172.21.15.155:0/699372246 >> v1:172.21.15.155:6790/0 conn(0x7f160c1457f0 legacy=0x7f160c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.096+0000 7f1612548640 1 -- 172.21.15.155:0/699372246 shutdown_connections 2024-03-29T11:35:04.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.096+0000 7f1612548640 1 -- 172.21.15.155:0/699372246 >> 172.21.15.155:0/699372246 conn(0x7f160c066a00 msgr2=0x7f160c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:04.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.099+0000 7f1612548640 1 -- 172.21.15.155:0/699372246 shutdown_connections 2024-03-29T11:35:04.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.100+0000 7f1612548640 1 -- 172.21.15.155:0/699372246 wait complete. 2024-03-29T11:35:04.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.101+0000 7f1612548640 10 monclient: build_initial_monmap 2024-03-29T11:35:04.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.101+0000 7f1612548640 10 monclient: monmap: 2024-03-29T11:35:04.100 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:04.100 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:04.100 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:35:04.100 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:35:04.100 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:04.100 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:04.100 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:04.100 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:35:04.101 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:35:04.101 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:04.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.101+0000 7f1612548640 1 librados: starting msgr at 2024-03-29T11:35:04.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.101+0000 7f1612548640 1 librados: starting objecter 2024-03-29T11:35:04.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.101+0000 7f1612548640 1 Processor -- start 2024-03-29T11:35:04.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.104+0000 7f1612548640 1 -- start start 2024-03-29T11:35:04.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.104+0000 7f1612548640 1 librados: setting wanted keys 2024-03-29T11:35:04.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.104+0000 7f1612548640 1 librados: calling monclient init 2024-03-29T11:35:04.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.104+0000 7f1612548640 10 monclient: init 2024-03-29T11:35:04.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.104+0000 7f1612548640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:04.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.104+0000 7f1612548640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:04.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.107+0000 7f1612548640 10 monclient(hunting): picked mon.noname-c con 0x7f160c0a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:04.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1612548640 10 monclient(hunting): picked mon.noname-a con 0x7f160c13f650 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:04.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1612548640 10 monclient(hunting): picked mon.noname-b con 0x7f160c1457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:04.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1612548640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c141b90 con 0x7f160c13f650 2024-03-29T11:35:04.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1612548640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c06e9d0 con 0x7f160c1457f0 2024-03-29T11:35:04.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1612548640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c141790 con 0x7f160c0a9d70 2024-03-29T11:35:04.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1612548640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:04.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1612548640 10 monclient(hunting): authenticate will time out at 886.235008s 2024-03-29T11:35:04.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f160bfff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f160c0a9d70 0x7f160c1421c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55847/0 (socket says 172.21.15.155:55847) 2024-03-29T11:35:04.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f160bfff640 1 -- 172.21.15.155:0/2917634723 learned_addr learned my addr 172.21.15.155:0/2917634723 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:04.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1608ff9640 10 client.?.objecter ms_handle_connect 0x7f160c0a9d70 2024-03-29T11:35:04.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1608ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754271218 0 0) 0x7f160c141790 con 0x7f160c0a9d70 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1608ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1608ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1608ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1608ff9640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.108+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f8004eb0 con 0x7f160c0a9d70 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.109+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1721408883 0 0) 0x7f15f8004eb0 con 0x7f160c0a9d70 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.109+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f160c141790 con 0x7f160c0a9d70 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.109+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1600001c00 con 0x7f160c0a9d70 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.109+0000 7f1608ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.109+0000 7f1608ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.109+0000 7f1608ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:04.108 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:04.109 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:04.109 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.109 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:04.109 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:04.109 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:04.109 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:04.109 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:04.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.109+0000 7f1608ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:04.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.109+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2072820958 0 0) 0x7f160c141790 con 0x7f160c0a9d70 2024-03-29T11:35:04.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.109+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 >> v1:172.21.15.183:6789/0 conn(0x7f160c1457f0 legacy=0x7f160c1404b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:04.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.111+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 >> v1:172.21.15.155:6789/0 conn(0x7f160c13f650 legacy=0x7f160c13fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:04.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:04.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 1 monclient: found mon.c 2024-03-29T11:35:04.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c0b1660 con 0x7f160c0a9d70 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1612548640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1612548640 10 monclient: _renew_subs 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1612548640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1612548640 1 -- 172.21.15.155:0/2917634723 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160c140bc0 con 0x7f160c0a9d70 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1612548640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1612548640 10 monclient: _renew_subs 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1612548640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1612548640 1 -- 172.21.15.155:0/2917634723 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f160c139ba0 con 0x7f160c0a9d70 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1612548640 1 librados: init done 2024-03-29T11:35:04.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1600004b00 con 0x7f160c0a9d70 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1600004d70 con 0x7f160c0a9d70 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.112+0000 7f1608ff9640 10 monclient: dump: 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:04.116 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.113+0000 7f1609ffb640 4 set_mon_vals no callback set 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.114+0000 7f15f27fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.115+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f1600005fc0 con 0x7f160c0a9d70 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.115+0000 7f15f27fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.115+0000 7f15f27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.115+0000 7f15f27fc640 1 -- 172.21.15.155:0/2917634723 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ec005080 con 0x7f160c0a9d70 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.115+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f1600084ba0 con 0x7f160c0a9d70 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.115+0000 7f160b7fe640 10 monclient: get_auth_request con 0x7f15f806dd00 auth_method 0 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.116+0000 7f1608ff9640 10 client.4250.objecter ms_dispatch 0x7f160c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.116+0000 7f1608ff9640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.116+0000 7f1608ff9640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.116+0000 7f1608ff9640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T11:35:04.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.116+0000 7f1608ff9640 10 client.4250.objecter ms_handle_connect 0x7f15f806dd00 2024-03-29T11:35:04.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.123+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1600050480 con 0x7f160c0a9d70 2024-03-29T11:35:04.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.123+0000 7f1608ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:04.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.123+0000 7f1608ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:04.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.435+0000 7f15f27fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:04.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.435+0000 7f15f27fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:04.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.435+0000 7f15f27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:35:04.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.435+0000 7f15f27fc640 1 -- 172.21.15.155:0/2917634723 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15ec005330 con 0x7f160c0a9d70 2024-03-29T11:35:04.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.437+0000 7f1608ff9640 1 -- 172.21.15.155:0/2917634723 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f16000523b0 con 0x7f160c0a9d70 2024-03-29T11:35:04.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.437+0000 7f1608ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:04.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.437+0000 7f1608ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:04.436 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:35:04.436 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:04.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.440+0000 7f15f27fc640 10 librados: watch_flush enter 2024-03-29T11:35:04.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.440+0000 7f15f27fc640 10 librados: watch_flush exit 2024-03-29T11:35:04.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.440+0000 7f15f27fc640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T11:35:04.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.440+0000 7f15f27fc640 10 client.4250.objecter successfully canceled tick 2024-03-29T11:35:04.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.440+0000 7f15f27fc640 1 -- 172.21.15.155:0/2917634723 >> v1:172.21.15.155:6816/1760814708 conn(0x7f15f806dd00 legacy=0x7f15f80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.440+0000 7f15f27fc640 10 monclient: shutdown 2024-03-29T11:35:04.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.440+0000 7f15f27fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:04.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.440+0000 7f15f27fc640 1 -- 172.21.15.155:0/2917634723 >> v1:172.21.15.155:6790/0 conn(0x7f160c0a9d70 legacy=0x7f160c1421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.441+0000 7f15f27fc640 1 -- 172.21.15.155:0/2917634723 shutdown_connections 2024-03-29T11:35:04.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.441+0000 7f15f27fc640 1 -- 172.21.15.155:0/2917634723 >> 172.21.15.155:0/2917634723 conn(0x7f160c066a00 msgr2=0x7f160c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:04.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.441+0000 7f15f27fc640 1 -- 172.21.15.155:0/2917634723 shutdown_connections 2024-03-29T11:35:04.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.441+0000 7f15f27fc640 1 -- 172.21.15.155:0/2917634723 wait complete. 2024-03-29T11:35:04.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:04.441+0000 7f15f27fc640 1 librados: shutdown 2024-03-29T11:35:04.457 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:11.263 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:11.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.423+0000 7fde96b8f640 10 monclient: get_monmap_and_config 2024-03-29T11:35:11.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.423+0000 7fde96b8f640 10 monclient: build_initial_monmap 2024-03-29T11:35:11.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.424+0000 7fde96b8f640 10 monclient: monmap: 2024-03-29T11:35:11.431 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:11.431 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:11.431 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:35:11.425433+0000 2024-03-29T11:35:11.431 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:35:11.425433+0000 2024-03-29T11:35:11.432 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:11.432 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:11.432 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:11.432 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:35:11.432 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:35:11.432 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:11.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.426+0000 7fde96b8f640 1 Processor -- start 2024-03-29T11:35:11.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.433+0000 7fde96b8f640 1 -- start start 2024-03-29T11:35:11.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.433+0000 7fde96b8f640 10 monclient: init 2024-03-29T11:35:11.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.434+0000 7fde96b8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:11.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.434+0000 7fde96b8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:11.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.434+0000 7fde96b8f640 10 monclient(hunting): picked mon.noname-b con 0x7fde900a9750 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:11.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.435+0000 7fde96b8f640 10 monclient(hunting): picked mon.noname-a con 0x7fde90149ac0 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:11.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.435+0000 7fde96b8f640 10 monclient(hunting): picked mon.noname-c con 0x7fde9014fe50 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:11.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.435+0000 7fde96b8f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde900aa4a0 con 0x7fde90149ac0 2024-03-29T11:35:11.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.435+0000 7fde96b8f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9014a680 con 0x7fde9014fe50 2024-03-29T11:35:11.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.435+0000 7fde96b8f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde900af290 con 0x7fde900a9750 2024-03-29T11:35:11.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.435+0000 7fde96b8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:11.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.435+0000 7fde96b8f640 10 monclient(hunting): authenticate will time out at 893.562128s 2024-03-29T11:35:11.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.439+0000 7fde9512c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fde9014fe50 0x7fde90152240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:40421/0 (socket says 172.21.15.155:40421) 2024-03-29T11:35:11.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.439+0000 7fde9512c640 1 -- 172.21.15.155:0/797232691 learned_addr learned my addr 172.21.15.155:0/797232691 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:11.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 1 -- 172.21.15.155:0/797232691 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303373794 0 0) 0x7fde9014a680 con 0x7fde9014fe50 2024-03-29T11:35:11.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:35:11.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 1 -- 172.21.15.155:0/797232691 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde80003520 con 0x7fde9014fe50 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 1 -- 172.21.15.155:0/797232691 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722805580 0 0) 0x7fde900aa4a0 con 0x7fde90149ac0 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 1 -- 172.21.15.155:0/797232691 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde9014a680 con 0x7fde90149ac0 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 1 -- 172.21.15.155:0/797232691 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1691549396 0 0) 0x7fde80003520 con 0x7fde9014fe50 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 1 -- 172.21.15.155:0/797232691 >> v1:172.21.15.155:6790/0 conn(0x7fde900a9750 legacy=0x7fde901436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 1 -- 172.21.15.155:0/797232691 >> v1:172.21.15.155:6789/0 conn(0x7fde90149ac0 legacy=0x7fde90149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:11.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 1 -- 172.21.15.155:0/797232691 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde900afb40 con 0x7fde9014fe50 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 1 -- 172.21.15.155:0/797232691 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fde88001a10 con 0x7fde9014fe50 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde96b8f640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde96b8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde8f7fe640 10 monclient: dump: 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:11.440 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:11.441 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:11.441 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.441 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:11.441 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:11.441 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:11.441 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:11.441 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:11.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.440+0000 7fde96b8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:11.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.441+0000 7fde8f7fe640 1 -- 172.21.15.155:0/797232691 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde88001bf0 con 0x7fde9014fe50 2024-03-29T11:35:11.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.441+0000 7fde8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:11.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.441+0000 7fde8f7fe640 1 -- 172.21.15.155:0/797232691 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fde88002000 con 0x7fde9014fe50 2024-03-29T11:35:11.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.441+0000 7fde8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.441+0000 7fde8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:11.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.441+0000 7fde8f7fe640 10 monclient: dump: 2024-03-29T11:35:11.442 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:11.443 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:11.443 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:11.443 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:11.443 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.443 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:11.443 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:11.443 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:11.443 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:11.443 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:11.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.442+0000 7fde96b8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:11.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.442+0000 7fde96b8f640 4 set_mon_vals no callback set 2024-03-29T11:35:11.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.444+0000 7fde96b8f640 10 monclient: shutdown 2024-03-29T11:35:11.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.444+0000 7fde96b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:11.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.444+0000 7fde96b8f640 1 -- 172.21.15.155:0/797232691 >> v1:172.21.15.183:6789/0 conn(0x7fde9014fe50 legacy=0x7fde90152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.445+0000 7fde96b8f640 1 -- 172.21.15.155:0/797232691 shutdown_connections 2024-03-29T11:35:11.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.445+0000 7fde96b8f640 1 -- 172.21.15.155:0/797232691 >> 172.21.15.155:0/797232691 conn(0x7fde900b35e0 msgr2=0x7fde900b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:11.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.445+0000 7fde96b8f640 1 -- 172.21.15.155:0/797232691 shutdown_connections 2024-03-29T11:35:11.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.445+0000 7fde96b8f640 1 -- 172.21.15.155:0/797232691 wait complete. 2024-03-29T11:35:11.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.446+0000 7fde96b8f640 10 monclient: build_initial_monmap 2024-03-29T11:35:11.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.446+0000 7fde96b8f640 10 monclient: monmap: 2024-03-29T11:35:11.445 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:11.445 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:11.445 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:35:11.445 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:35:11.445 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:11.445 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:11.445 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:11.445 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:35:11.446 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:35:11.446 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:11.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.446+0000 7fde96b8f640 1 librados: starting msgr at 2024-03-29T11:35:11.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.446+0000 7fde96b8f640 1 librados: starting objecter 2024-03-29T11:35:11.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.446+0000 7fde96b8f640 1 Processor -- start 2024-03-29T11:35:11.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.447+0000 7fde96b8f640 1 -- start start 2024-03-29T11:35:11.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.447+0000 7fde96b8f640 1 librados: setting wanted keys 2024-03-29T11:35:11.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.447+0000 7fde96b8f640 1 librados: calling monclient init 2024-03-29T11:35:11.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.447+0000 7fde96b8f640 10 monclient: init 2024-03-29T11:35:11.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.447+0000 7fde96b8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:11.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.447+0000 7fde96b8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:11.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.447+0000 7fde96b8f640 10 monclient(hunting): picked mon.noname-c con 0x7fde900a9750 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:11.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.449+0000 7fde96b8f640 10 monclient(hunting): picked mon.noname-b con 0x7fde90149ac0 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:11.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.449+0000 7fde96b8f640 10 monclient(hunting): picked mon.noname-a con 0x7fde9014fe50 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:11.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.449+0000 7fde96b8f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde900711c0 con 0x7fde9014fe50 2024-03-29T11:35:11.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.449+0000 7fde96b8f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde90070cc0 con 0x7fde90149ac0 2024-03-29T11:35:11.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.449+0000 7fde96b8f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9006ff70 con 0x7fde900a9750 2024-03-29T11:35:11.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.449+0000 7fde96b8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:11.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.449+0000 7fde96b8f640 10 monclient(hunting): authenticate will time out at 893.576625s 2024-03-29T11:35:11.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.454+0000 7fde9492b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fde900a9750 0x7fde901dee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:44507/0 (socket says 172.21.15.155:44507) 2024-03-29T11:35:11.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.454+0000 7fde9492b640 1 -- 172.21.15.155:0/1699551083 learned_addr learned my addr 172.21.15.155:0/1699551083 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:11.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.454+0000 7fde8d7fa640 10 client.?.objecter ms_handle_connect 0x7fde900a9750 2024-03-29T11:35:11.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.454+0000 7fde8d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:11.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.454+0000 7fde8d7fa640 10 client.?.objecter ms_handle_connect 0x7fde9014fe50 2024-03-29T11:35:11.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.454+0000 7fde8d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:11.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.454+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715272400 0 0) 0x7fde900711c0 con 0x7fde9014fe50 2024-03-29T11:35:11.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.454+0000 7fde8d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.454+0000 7fde8d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.454+0000 7fde8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.455+0000 7fde8d7fa640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:35:11.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.455+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde74004eb0 con 0x7fde9014fe50 2024-03-29T11:35:11.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.455+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188627773 0 0) 0x7fde9006ff70 con 0x7fde900a9750 2024-03-29T11:35:11.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.455+0000 7fde8d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.455+0000 7fde8d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.455+0000 7fde8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.455+0000 7fde8d7fa640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde900711c0 con 0x7fde900a9750 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4181629432 0 0) 0x7fde74004eb0 con 0x7fde9014fe50 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde9006ff70 con 0x7fde9014fe50 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fde88001f00 con 0x7fde9014fe50 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 10 monclient(hunting): dump: 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:11.459 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 10 client.?.objecter ms_handle_connect 0x7fde90149ac0 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4245782031 0 0) 0x7fde9006ff70 con 0x7fde9014fe50 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 >> v1:172.21.15.155:6790/0 conn(0x7fde900a9750 legacy=0x7fde901dee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 >> v1:172.21.15.183:6789/0 conn(0x7fde90149ac0 legacy=0x7fde90140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 1 monclient: found mon.a 2024-03-29T11:35:11.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde900713c0 con 0x7fde9014fe50 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde96b8f640 5 monclient: authenticate success, global_id 4254 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde96b8f640 10 monclient: _renew_subs 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde96b8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde96b8f640 1 -- 172.21.15.155:0/1699551083 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde90071610 con 0x7fde9014fe50 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde96b8f640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde96b8f640 10 monclient: _renew_subs 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde96b8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde96b8f640 1 -- 172.21.15.155:0/1699551083 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde900709c0 con 0x7fde9014fe50 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde96b8f640 1 librados: init done 2024-03-29T11:35:11.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde880013e0 con 0x7fde9014fe50 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.456+0000 7fde8effd640 4 set_mon_vals no callback set 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.458+0000 7fde96b8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.458+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fde88004f70 con 0x7fde9014fe50 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.458+0000 7fde96b8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.458+0000 7fde96b8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.458+0000 7fde96b8f640 1 -- 172.21.15.155:0/1699551083 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde70005080 con 0x7fde9014fe50 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.458+0000 7fde8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.459+0000 7fde8d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.459+0000 7fde8d7fa640 10 monclient: dump: 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:11.462 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:11.463 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:11.463 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.463 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:11.463 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:11.463 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:11.463 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:11.463 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:11.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.460+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fde8801af40 con 0x7fde9014fe50 2024-03-29T11:35:11.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.461+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fde88084dd0 con 0x7fde9014fe50 2024-03-29T11:35:11.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.461+0000 7fde8d7fa640 10 client.4254.objecter ms_dispatch 0x7fde90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:11.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.461+0000 7fde8d7fa640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:11.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.461+0000 7fde8d7fa640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:11.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.461+0000 7fde8d7fa640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-29T11:35:11.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.461+0000 7fde9492b640 10 monclient: get_auth_request con 0x7fde7406dac0 auth_method 0 2024-03-29T11:35:11.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.462+0000 7fde8d7fa640 10 client.4254.objecter ms_handle_connect 0x7fde7406dac0 2024-03-29T11:35:11.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.469+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde880504e0 con 0x7fde9014fe50 2024-03-29T11:35:11.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.469+0000 7fde8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:11.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.469+0000 7fde8d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:11.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.787+0000 7fde96b8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:11.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.787+0000 7fde96b8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:11.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.787+0000 7fde96b8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:11.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.787+0000 7fde96b8f640 1 -- 172.21.15.155:0/1699551083 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde70005330 con 0x7fde9014fe50 2024-03-29T11:35:11.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.791+0000 7fde8d7fa640 1 -- 172.21.15.155:0/1699551083 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fde880525e0 con 0x7fde9014fe50 2024-03-29T11:35:11.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.791+0000 7fde8d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:11.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.791+0000 7fde8d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:11.791 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:35:11.791 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:11.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.795+0000 7fde96b8f640 10 librados: watch_flush enter 2024-03-29T11:35:11.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.795+0000 7fde96b8f640 10 librados: watch_flush exit 2024-03-29T11:35:11.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.795+0000 7fde96b8f640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-29T11:35:11.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.795+0000 7fde96b8f640 10 client.4254.objecter successfully canceled tick 2024-03-29T11:35:11.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.795+0000 7fde96b8f640 1 -- 172.21.15.155:0/1699551083 >> v1:172.21.15.155:6816/1760814708 conn(0x7fde7406dac0 legacy=0x7fde7406ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.795+0000 7fde96b8f640 10 monclient: shutdown 2024-03-29T11:35:11.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.795+0000 7fde96b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:11.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.795+0000 7fde96b8f640 1 -- 172.21.15.155:0/1699551083 >> v1:172.21.15.155:6789/0 conn(0x7fde9014fe50 legacy=0x7fde901e71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.796+0000 7fde96b8f640 1 -- 172.21.15.155:0/1699551083 shutdown_connections 2024-03-29T11:35:11.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.796+0000 7fde96b8f640 1 -- 172.21.15.155:0/1699551083 >> 172.21.15.155:0/1699551083 conn(0x7fde900b35e0 msgr2=0x7fde90143af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:11.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.796+0000 7fde96b8f640 1 -- 172.21.15.155:0/1699551083 shutdown_connections 2024-03-29T11:35:11.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.796+0000 7fde96b8f640 1 -- 172.21.15.155:0/1699551083 wait complete. 2024-03-29T11:35:11.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:11.796+0000 7fde96b8f640 1 librados: shutdown 2024-03-29T11:35:11.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:18.617 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:18.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.771+0000 7fa9166c0640 10 monclient: get_monmap_and_config 2024-03-29T11:35:18.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.771+0000 7fa9166c0640 10 monclient: build_initial_monmap 2024-03-29T11:35:18.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.771+0000 7fa9166c0640 10 monclient: monmap: 2024-03-29T11:35:18.777 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:18.777 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:18.777 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:35:18.773216+0000 2024-03-29T11:35:18.777 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:35:18.773216+0000 2024-03-29T11:35:18.777 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.772+0000 7fa9166c0640 1 Processor -- start 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.772+0000 7fa9166c0640 1 -- start start 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.772+0000 7fa9166c0640 10 monclient: init 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.773+0000 7fa9166c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.773+0000 7fa9166c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.773+0000 7fa9166c0640 10 monclient(hunting): picked mon.noname-b con 0x7fa9100ad6c0 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.773+0000 7fa9166c0640 10 monclient(hunting): picked mon.noname-a con 0x7fa9100af6c0 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.773+0000 7fa9166c0640 10 monclient(hunting): picked mon.noname-c con 0x7fa91014fdc0 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:18.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.773+0000 7fa9166c0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9100a94e0 con 0x7fa9100af6c0 2024-03-29T11:35:18.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.773+0000 7fa9166c0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9100af3a0 con 0x7fa91014fdc0 2024-03-29T11:35:18.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.773+0000 7fa9166c0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9100afb60 con 0x7fa9100ad6c0 2024-03-29T11:35:18.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.773+0000 7fa9166c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:18.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.773+0000 7fa9166c0640 10 monclient(hunting): authenticate will time out at 900.900696s 2024-03-29T11:35:18.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.778+0000 7fa90f7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fa9100af6c0 0x7fa910152980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:44321/0 (socket says 172.21.15.155:44321) 2024-03-29T11:35:18.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.778+0000 7fa90f7fe640 1 -- 172.21.15.155:0/3168028589 learned_addr learned my addr 172.21.15.155:0/3168028589 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:18.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.778+0000 7fa90effd640 1 -- 172.21.15.155:0/3168028589 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940250127 0 0) 0x7fa9100a94e0 con 0x7fa9100af6c0 2024-03-29T11:35:18.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.778+0000 7fa90effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:18.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.778+0000 7fa90effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.778+0000 7fa90effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.778+0000 7fa90effd640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.778+0000 7fa90effd640 1 -- 172.21.15.155:0/3168028589 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f8003500 con 0x7fa9100af6c0 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.779+0000 7fa90effd640 1 -- 172.21.15.155:0/3168028589 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1035754853 0 0) 0x7fa8f8003500 con 0x7fa9100af6c0 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.779+0000 7fa90effd640 1 -- 172.21.15.155:0/3168028589 >> v1:172.21.15.155:6790/0 conn(0x7fa9100ad6c0 legacy=0x7fa91014f6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 1 -- 172.21.15.155:0/3168028589 >> v1:172.21.15.183:6789/0 conn(0x7fa91014fdc0 legacy=0x7fa91015a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 1 monclient: found mon.noname-a 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 1 -- 172.21.15.155:0/3168028589 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9100aab80 con 0x7fa9100af6c0 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 10 monclient: _finish_auth 0 2024-03-29T11:35:18.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 1 -- 172.21.15.155:0/3168028589 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa908001670 con 0x7fa9100af6c0 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa90effd640 10 monclient: dump: 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:18.783 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:18.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa9166c0640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:35:18.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.780+0000 7fa9166c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:18.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.784+0000 7fa90effd640 1 -- 172.21.15.155:0/3168028589 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9080018d0 con 0x7fa9100af6c0 2024-03-29T11:35:18.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.784+0000 7fa90effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:18.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.784+0000 7fa90effd640 1 -- 172.21.15.155:0/3168028589 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa908001ce0 con 0x7fa9100af6c0 2024-03-29T11:35:18.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.784+0000 7fa90effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:18.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.784+0000 7fa90effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.784+0000 7fa90effd640 10 monclient: dump: 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.784+0000 7fa9166c0640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.784+0000 7fa9166c0640 4 set_mon_vals no callback set 2024-03-29T11:35:18.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.785+0000 7fa9166c0640 10 monclient: shutdown 2024-03-29T11:35:18.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.785+0000 7fa9166c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:18.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.785+0000 7fa9166c0640 1 -- 172.21.15.155:0/3168028589 >> v1:172.21.15.155:6789/0 conn(0x7fa9100af6c0 legacy=0x7fa910152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.785+0000 7fa9166c0640 1 -- 172.21.15.155:0/3168028589 shutdown_connections 2024-03-29T11:35:18.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.785+0000 7fa9166c0640 1 -- 172.21.15.155:0/3168028589 >> 172.21.15.155:0/3168028589 conn(0x7fa9100b35e0 msgr2=0x7fa9100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:18.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.786+0000 7fa9166c0640 1 -- 172.21.15.155:0/3168028589 shutdown_connections 2024-03-29T11:35:18.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.786+0000 7fa9166c0640 1 -- 172.21.15.155:0/3168028589 wait complete. 2024-03-29T11:35:18.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.787+0000 7fa9166c0640 10 monclient: build_initial_monmap 2024-03-29T11:35:18.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.787+0000 7fa9166c0640 10 monclient: monmap: 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.787+0000 7fa9166c0640 1 librados: starting msgr at 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.787+0000 7fa9166c0640 1 librados: starting objecter 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.787+0000 7fa9166c0640 1 Processor -- start 2024-03-29T11:35:18.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.788+0000 7fa9166c0640 1 -- start start 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.788+0000 7fa9166c0640 1 librados: setting wanted keys 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.788+0000 7fa9166c0640 1 librados: calling monclient init 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.788+0000 7fa9166c0640 10 monclient: init 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.788+0000 7fa9166c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.788+0000 7fa9166c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.788+0000 7fa9166c0640 10 monclient(hunting): picked mon.noname-c con 0x7fa9100ad6c0 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.788+0000 7fa9166c0640 10 monclient(hunting): picked mon.noname-a con 0x7fa9100af6c0 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.788+0000 7fa9166c0640 10 monclient(hunting): picked mon.noname-b con 0x7fa91014fdc0 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.788+0000 7fa9166c0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91007b930 con 0x7fa9100af6c0 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.789+0000 7fa9166c0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa910071c60 con 0x7fa91014fdc0 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.789+0000 7fa9166c0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa910071910 con 0x7fa9100ad6c0 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.789+0000 7fa9166c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:18.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.789+0000 7fa9166c0640 10 monclient(hunting): authenticate will time out at 900.915897s 2024-03-29T11:35:18.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa914c5d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fa91014fdc0 0x7fa9101ff900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:56961/0 (socket says 172.21.15.155:56961) 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa914c5d640 1 -- 172.21.15.155:0/1389760444 learned_addr learned my addr 172.21.15.155:0/1389760444 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 10 client.?.objecter ms_handle_connect 0x7fa91014fdc0 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863255489 0 0) 0x7fa910071c60 con 0x7fa91014fdc0 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8fc004eb0 con 0x7fa91014fdc0 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2741015302 0 0) 0x7fa8fc004eb0 con 0x7fa91014fdc0 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa910071c60 con 0x7fa91014fdc0 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa904001870 con 0x7fa91014fdc0 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:18.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 10 monclient(hunting): dump: 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:18.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.795+0000 7fa90cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:18.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.797+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 795867171 0 0) 0x7fa910071c60 con 0x7fa91014fdc0 2024-03-29T11:35:18.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.797+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 >> v1:172.21.15.155:6790/0 conn(0x7fa9100ad6c0 legacy=0x7fa910071200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.797+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 >> v1:172.21.15.155:6789/0 conn(0x7fa9100af6c0 legacy=0x7fa9101f4e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:18.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa90cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:18.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa90cff9640 1 monclient: found mon.b 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa90cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa90cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91014e890 con 0x7fa91014fdc0 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa90cff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa90cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa90cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa9166c0640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa9166c0640 10 monclient: _renew_subs 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa9166c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa9166c0640 1 -- 172.21.15.155:0/1389760444 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91014eae0 con 0x7fa91014fdc0 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa9166c0640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa9166c0640 10 monclient: _renew_subs 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa9166c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:35:18.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa9166c0640 1 -- 172.21.15.155:0/1389760444 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9101f56f0 con 0x7fa91014fdc0 2024-03-29T11:35:18.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.798+0000 7fa9166c0640 1 librados: init done 2024-03-29T11:35:18.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.800+0000 7fa9166c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:18.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.800+0000 7fa9166c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:18.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.800+0000 7fa9166c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:35:18.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.800+0000 7fa9166c0640 1 -- 172.21.15.155:0/1389760444 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa910071aa0 con 0x7fa91014fdc0 2024-03-29T11:35:18.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.802+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9040019d0 con 0x7fa91014fdc0 2024-03-29T11:35:18.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.802+0000 7fa90cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:18.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.802+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa904003870 con 0x7fa91014fdc0 2024-03-29T11:35:18.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.802+0000 7fa90cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:18.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.802+0000 7fa90cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:18.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.802+0000 7fa90cff9640 10 monclient: dump: 2024-03-29T11:35:18.800 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:18.800 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:18.801 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:18.801 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:18.801 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:18.801 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:18.801 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:18.801 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:18.801 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:18.801 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:18.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.802+0000 7fa90dffb640 4 set_mon_vals no callback set 2024-03-29T11:35:18.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.803+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fa904003a90 con 0x7fa91014fdc0 2024-03-29T11:35:18.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.804+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fa904003690 con 0x7fa91014fdc0 2024-03-29T11:35:18.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.804+0000 7fa90cff9640 10 client.4258.objecter ms_dispatch 0x7fa910000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:18.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.804+0000 7fa90cff9640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:18.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.804+0000 7fa90cff9640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:18.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.804+0000 7fa90cff9640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-29T11:35:18.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.804+0000 7fa90ffff640 10 monclient: get_auth_request con 0x7fa8fc06dd50 auth_method 0 2024-03-29T11:35:18.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.805+0000 7fa90cff9640 10 client.4258.objecter ms_handle_connect 0x7fa8fc06dd50 2024-03-29T11:35:18.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.810+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa904050950 con 0x7fa91014fdc0 2024-03-29T11:35:18.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.811+0000 7fa90cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:18.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:18.811+0000 7fa90cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:19.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.135+0000 7fa9166c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:19.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.135+0000 7fa9166c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:19.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.136+0000 7fa9166c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:35:19.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.136+0000 7fa9166c0640 1 -- 172.21.15.155:0/1389760444 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9101f5990 con 0x7fa91014fdc0 2024-03-29T11:35:19.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.137+0000 7fa90cff9640 1 -- 172.21.15.155:0/1389760444 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fa904050af0 con 0x7fa91014fdc0 2024-03-29T11:35:19.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.137+0000 7fa90cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:19.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.137+0000 7fa90cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:19.135 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:35:19.135 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:19.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.140+0000 7fa9166c0640 10 librados: watch_flush enter 2024-03-29T11:35:19.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.142+0000 7fa9166c0640 10 librados: watch_flush exit 2024-03-29T11:35:19.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.142+0000 7fa9166c0640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-29T11:35:19.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.142+0000 7fa9166c0640 10 client.4258.objecter successfully canceled tick 2024-03-29T11:35:19.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.142+0000 7fa9166c0640 1 -- 172.21.15.155:0/1389760444 >> v1:172.21.15.155:6816/1760814708 conn(0x7fa8fc06dd50 legacy=0x7fa8fc070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:19.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.142+0000 7fa9166c0640 10 monclient: shutdown 2024-03-29T11:35:19.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.142+0000 7fa9166c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:19.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.142+0000 7fa9166c0640 1 -- 172.21.15.155:0/1389760444 >> v1:172.21.15.183:6789/0 conn(0x7fa91014fdc0 legacy=0x7fa9101ff900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:19.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.143+0000 7fa9166c0640 1 -- 172.21.15.155:0/1389760444 shutdown_connections 2024-03-29T11:35:19.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.143+0000 7fa9166c0640 1 -- 172.21.15.155:0/1389760444 >> 172.21.15.155:0/1389760444 conn(0x7fa9100b35e0 msgr2=0x7fa9100a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:19.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.145+0000 7fa9166c0640 1 -- 172.21.15.155:0/1389760444 shutdown_connections 2024-03-29T11:35:19.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.145+0000 7fa9166c0640 1 -- 172.21.15.155:0/1389760444 wait complete. 2024-03-29T11:35:19.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:19.145+0000 7fa9166c0640 1 librados: shutdown 2024-03-29T11:35:19.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:25.967 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:26.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.125+0000 7f4f41471640 10 monclient: get_monmap_and_config 2024-03-29T11:35:26.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.125+0000 7f4f41471640 10 monclient: build_initial_monmap 2024-03-29T11:35:26.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.125+0000 7f4f41471640 10 monclient: monmap: 2024-03-29T11:35:26.123 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:26.123 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:26.123 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:35:26.126070+0000 2024-03-29T11:35:26.123 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:35:26.126070+0000 2024-03-29T11:35:26.123 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:26.123 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:26.123 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:26.124 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:35:26.124 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:35:26.124 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:26.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.128+0000 7f4f41471640 1 Processor -- start 2024-03-29T11:35:26.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.128+0000 7f4f41471640 1 -- start start 2024-03-29T11:35:26.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.128+0000 7f4f41471640 10 monclient: init 2024-03-29T11:35:26.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.129+0000 7f4f41471640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:26.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.129+0000 7f4f41471640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:26.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.129+0000 7f4f41471640 10 monclient(hunting): picked mon.noname-b con 0x7f4f3c0b2800 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:26.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.129+0000 7f4f41471640 10 monclient(hunting): picked mon.noname-a con 0x7f4f3c142b20 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:26.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.129+0000 7f4f41471640 10 monclient(hunting): picked mon.noname-c con 0x7f4f3c13c040 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:26.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.129+0000 7f4f41471640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c0aad10 con 0x7f4f3c142b20 2024-03-29T11:35:26.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.129+0000 7f4f41471640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c141f30 con 0x7f4f3c13c040 2024-03-29T11:35:26.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.129+0000 7f4f41471640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c0aaf60 con 0x7f4f3c0b2800 2024-03-29T11:35:26.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.129+0000 7f4f41471640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:26.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.129+0000 7f4f41471640 10 monclient(hunting): authenticate will time out at 908.255035s 2024-03-29T11:35:26.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.133+0000 7f4f3bfff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f4f3c0b2800 0x7f4f3c142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:40771/0 (socket says 172.21.15.155:40771) 2024-03-29T11:35:26.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.133+0000 7f4f3bfff640 1 -- 172.21.15.155:0/1287026190 learned_addr learned my addr 172.21.15.155:0/1287026190 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:26.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.136+0000 7f4f3affd640 1 -- 172.21.15.155:0/1287026190 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448633597 0 0) 0x7f4f3c0aaf60 con 0x7f4f3c0b2800 2024-03-29T11:35:26.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.136+0000 7f4f3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:26.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.136+0000 7f4f3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:26.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.136+0000 7f4f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:26.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.136+0000 7f4f3affd640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:35:26.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.136+0000 7f4f3affd640 1 -- 172.21.15.155:0/1287026190 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f24003520 con 0x7f4f3c0b2800 2024-03-29T11:35:26.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 1 -- 172.21.15.155:0/1287026190 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238783113 0 0) 0x7f4f3c0aad10 con 0x7f4f3c142b20 2024-03-29T11:35:26.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:26.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:26.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:26.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:35:26.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 1 -- 172.21.15.155:0/1287026190 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f3c0aaf60 con 0x7f4f3c142b20 2024-03-29T11:35:26.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 1 -- 172.21.15.155:0/1287026190 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2188089313 0 0) 0x7f4f24003520 con 0x7f4f3c0b2800 2024-03-29T11:35:26.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 1 -- 172.21.15.155:0/1287026190 >> v1:172.21.15.183:6789/0 conn(0x7f4f3c13c040 legacy=0x7f4f3c14d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:26.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 1 -- 172.21.15.155:0/1287026190 >> v1:172.21.15.155:6789/0 conn(0x7f4f3c142b20 legacy=0x7f4f3c13b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 1 monclient: found mon.noname-b 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 1 -- 172.21.15.155:0/1287026190 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c0b3900 con 0x7f4f3c0b2800 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 1 -- 172.21.15.155:0/1287026190 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4f2c001610 con 0x7f4f3c0b2800 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.137+0000 7f4f3affd640 10 monclient: dump: 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:26.136 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:26.137 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:26.137 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:26.137 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:26.137 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:26.137 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:26.137 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:26.137 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:26.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.138+0000 7f4f3affd640 1 -- 172.21.15.155:0/1287026190 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f2c001770 con 0x7f4f3c0b2800 2024-03-29T11:35:26.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.138+0000 7f4f3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:26.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.138+0000 7f4f3affd640 1 -- 172.21.15.155:0/1287026190 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4f2c001bc0 con 0x7f4f3c0b2800 2024-03-29T11:35:26.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.138+0000 7f4f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:26.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.138+0000 7f4f3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:26.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.138+0000 7f4f3affd640 10 monclient: dump: 2024-03-29T11:35:26.138 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:26.138 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:26.138 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:26.138 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.138+0000 7f4f41471640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.138+0000 7f4f41471640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.138+0000 7f4f41471640 4 set_mon_vals no callback set 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.139+0000 7f4f41471640 10 monclient: shutdown 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.139+0000 7f4f41471640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.139+0000 7f4f41471640 1 -- 172.21.15.155:0/1287026190 >> v1:172.21.15.155:6790/0 conn(0x7f4f3c0b2800 legacy=0x7f4f3c142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.140+0000 7f4f41471640 1 -- 172.21.15.155:0/1287026190 shutdown_connections 2024-03-29T11:35:26.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.140+0000 7f4f41471640 1 -- 172.21.15.155:0/1287026190 >> 172.21.15.155:0/1287026190 conn(0x7f4f3c06d0e0 msgr2=0x7f4f3c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:26.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.142+0000 7f4f41471640 1 -- 172.21.15.155:0/1287026190 shutdown_connections 2024-03-29T11:35:26.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.142+0000 7f4f41471640 1 -- 172.21.15.155:0/1287026190 wait complete. 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.142+0000 7f4f41471640 10 monclient: build_initial_monmap 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.142+0000 7f4f41471640 10 monclient: monmap: 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.143+0000 7f4f41471640 1 librados: starting msgr at 2024-03-29T11:35:26.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.143+0000 7f4f41471640 1 librados: starting objecter 2024-03-29T11:35:26.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.143+0000 7f4f41471640 1 Processor -- start 2024-03-29T11:35:26.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.143+0000 7f4f41471640 1 -- start start 2024-03-29T11:35:26.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.143+0000 7f4f41471640 1 librados: setting wanted keys 2024-03-29T11:35:26.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.143+0000 7f4f41471640 1 librados: calling monclient init 2024-03-29T11:35:26.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.143+0000 7f4f41471640 10 monclient: init 2024-03-29T11:35:26.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.144+0000 7f4f41471640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:26.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.144+0000 7f4f41471640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:26.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.144+0000 7f4f41471640 10 monclient(hunting): picked mon.noname-a con 0x7f4f3c0b2800 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:26.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.144+0000 7f4f41471640 10 monclient(hunting): picked mon.noname-c con 0x7f4f3c13c040 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:26.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.144+0000 7f4f41471640 10 monclient(hunting): picked mon.noname-b con 0x7f4f3c142b20 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:26.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.144+0000 7f4f41471640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c13ece0 con 0x7f4f3c0b2800 2024-03-29T11:35:26.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.144+0000 7f4f41471640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c13dc40 con 0x7f4f3c142b20 2024-03-29T11:35:26.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.144+0000 7f4f41471640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c13ef10 con 0x7f4f3c13c040 2024-03-29T11:35:26.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.144+0000 7f4f41471640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:26.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.144+0000 7f4f41471640 10 monclient(hunting): authenticate will time out at 908.270504s 2024-03-29T11:35:26.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.148+0000 7f4f3bfff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f4f3c0b2800 0x7f4f3c13d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:42965/0 (socket says 172.21.15.155:42965) 2024-03-29T11:35:26.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.148+0000 7f4f3bfff640 1 -- 172.21.15.155:0/3557816719 learned_addr learned my addr 172.21.15.155:0/3557816719 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:26.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.148+0000 7f4f38ff9640 10 client.?.objecter ms_handle_connect 0x7f4f3c0b2800 2024-03-29T11:35:26.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.148+0000 7f4f38ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:26.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.149+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028146449 0 0) 0x7f4f3c13ece0 con 0x7f4f3c0b2800 2024-03-29T11:35:26.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.149+0000 7f4f38ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:26.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.149+0000 7f4f38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:26.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.149+0000 7f4f38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:26.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.149+0000 7f4f38ff9640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:35:26.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.149+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f28004eb0 con 0x7f4f3c0b2800 2024-03-29T11:35:26.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 523297460 0 0) 0x7f4f28004eb0 con 0x7f4f3c0b2800 2024-03-29T11:35:26.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f28005300 con 0x7f4f3c0b2800 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4f2c001bc0 con 0x7f4f3c0b2800 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:26.151 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 client.?.objecter ms_handle_connect 0x7f4f3c142b20 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4284986877 0 0) 0x7f4f28005300 con 0x7f4f3c0b2800 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 >> v1:172.21.15.155:6790/0 conn(0x7f4f3c13c040 legacy=0x7f4f3c132d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 >> v1:172.21.15.183:6789/0 conn(0x7f4f3c142b20 legacy=0x7f4f3c1d2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 1 monclient: found mon.a 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c13c8b0 con 0x7f4f3c0b2800 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:26.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f2c004a90 con 0x7f4f3c0b2800 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4f2c001bc0 con 0x7f4f3c0b2800 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.150+0000 7f4f38ff9640 10 monclient: dump: 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:26.153 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:26.154 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:26.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.151+0000 7f4f41471640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:35:26.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.152+0000 7f4f41471640 10 monclient: _renew_subs 2024-03-29T11:35:26.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.152+0000 7f4f41471640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:26.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.152+0000 7f4f41471640 1 -- 172.21.15.155:0/3557816719 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3c0b13e0 con 0x7f4f3c0b2800 2024-03-29T11:35:26.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.152+0000 7f4f41471640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:26.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.152+0000 7f4f41471640 10 monclient: _renew_subs 2024-03-29T11:35:26.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.152+0000 7f4f41471640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:26.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.152+0000 7f4f39ffb640 4 set_mon_vals no callback set 2024-03-29T11:35:26.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.152+0000 7f4f41471640 1 -- 172.21.15.155:0/3557816719 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c133650 con 0x7f4f3c0b2800 2024-03-29T11:35:26.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.152+0000 7f4f41471640 1 librados: init done 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.155+0000 7f4f41471640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.155+0000 7f4f41471640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.155+0000 7f4f41471640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.155+0000 7f4f41471640 1 -- 172.21.15.155:0/3557816719 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c04f730 con 0x7f4f3c0b2800 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.156+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f4f2c01abb0 con 0x7f4f3c0b2800 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.157+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f4f2c0546f0 con 0x7f4f3c0b2800 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.157+0000 7f4f38ff9640 10 client.4263.objecter ms_dispatch 0x7f4f3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.157+0000 7f4f38ff9640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.157+0000 7f4f38ff9640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.157+0000 7f4f38ff9640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.158+0000 7f4f3b7fe640 10 monclient: get_auth_request con 0x7f4f2806e040 auth_method 0 2024-03-29T11:35:26.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.158+0000 7f4f38ff9640 10 client.4263.objecter ms_handle_connect 0x7f4f2806e040 2024-03-29T11:35:26.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.166+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f2c050990 con 0x7f4f3c0b2800 2024-03-29T11:35:26.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.166+0000 7f4f38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:26.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.166+0000 7f4f38ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:26.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.501+0000 7f4f41471640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:26.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.501+0000 7f4f41471640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:26.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.501+0000 7f4f41471640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:26.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.501+0000 7f4f41471640 1 -- 172.21.15.155:0/3557816719 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f3c074d80 con 0x7f4f3c0b2800 2024-03-29T11:35:26.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.502+0000 7f4f38ff9640 1 -- 172.21.15.155:0/3557816719 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f4f2c050b30 con 0x7f4f3c0b2800 2024-03-29T11:35:26.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.502+0000 7f4f38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:26.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.502+0000 7f4f38ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:26.502 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:35:26.502 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:26.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.507+0000 7f4f41471640 10 librados: watch_flush enter 2024-03-29T11:35:26.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.509+0000 7f4f41471640 10 librados: watch_flush exit 2024-03-29T11:35:26.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.509+0000 7f4f41471640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T11:35:26.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.509+0000 7f4f41471640 10 client.4263.objecter successfully canceled tick 2024-03-29T11:35:26.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.509+0000 7f4f41471640 1 -- 172.21.15.155:0/3557816719 >> v1:172.21.15.155:6816/1760814708 conn(0x7f4f2806e040 legacy=0x7f4f28070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:26.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.509+0000 7f4f41471640 10 monclient: shutdown 2024-03-29T11:35:26.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.509+0000 7f4f41471640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:26.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.509+0000 7f4f41471640 1 -- 172.21.15.155:0/3557816719 >> v1:172.21.15.155:6789/0 conn(0x7f4f3c0b2800 legacy=0x7f4f3c13d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:26.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.509+0000 7f4f41471640 1 -- 172.21.15.155:0/3557816719 shutdown_connections 2024-03-29T11:35:26.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.509+0000 7f4f41471640 1 -- 172.21.15.155:0/3557816719 >> 172.21.15.155:0/3557816719 conn(0x7f4f3c06d0e0 msgr2=0x7f4f3c142410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:26.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.509+0000 7f4f41471640 1 -- 172.21.15.155:0/3557816719 shutdown_connections 2024-03-29T11:35:26.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.509+0000 7f4f41471640 1 -- 172.21.15.155:0/3557816719 wait complete. 2024-03-29T11:35:26.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:26.511+0000 7f4f41471640 1 librados: shutdown 2024-03-29T11:35:26.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:33.331 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:33.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.476+0000 7f92229eb640 10 monclient: get_monmap_and_config 2024-03-29T11:35:33.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.476+0000 7f92229eb640 10 monclient: build_initial_monmap 2024-03-29T11:35:33.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.477+0000 7f92229eb640 10 monclient: monmap: 2024-03-29T11:35:33.475 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:33.475 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:33.475 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:35:33.477420+0000 2024-03-29T11:35:33.475 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:35:33.477420+0000 2024-03-29T11:35:33.475 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:33.475 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:33.475 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:33.475 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:35:33.476 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:35:33.476 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:33.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.478+0000 7f92229eb640 1 Processor -- start 2024-03-29T11:35:33.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.478+0000 7f92229eb640 1 -- start start 2024-03-29T11:35:33.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.478+0000 7f92229eb640 10 monclient: init 2024-03-29T11:35:33.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.479+0000 7f92229eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:33.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.479+0000 7f92229eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:33.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.479+0000 7f92229eb640 10 monclient(hunting): picked mon.noname-b con 0x7f9214006890 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:33.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.479+0000 7f92229eb640 10 monclient(hunting): picked mon.noname-a con 0x7f9214006c70 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:33.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.479+0000 7f92229eb640 10 monclient(hunting): picked mon.noname-c con 0x7f921402b4c0 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:33.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.479+0000 7f92229eb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921402c830 con 0x7f9214006c70 2024-03-29T11:35:33.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.479+0000 7f92229eb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921402afa0 con 0x7f921402b4c0 2024-03-29T11:35:33.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.479+0000 7f92229eb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921402aba0 con 0x7f9214006890 2024-03-29T11:35:33.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.479+0000 7f92229eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:33.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.479+0000 7f92229eb640 10 monclient(hunting): authenticate will time out at 915.605592s 2024-03-29T11:35:33.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.483+0000 7f92219e9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f9214006890 0x7f92140c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:59897/0 (socket says 172.21.15.155:59897) 2024-03-29T11:35:33.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.483+0000 7f92219e9640 1 -- 172.21.15.155:0/1500294489 learned_addr learned my addr 172.21.15.155:0/1500294489 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:33.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.486+0000 7f92209e7640 1 -- 172.21.15.155:0/1500294489 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513281009 0 0) 0x7f921402afa0 con 0x7f921402b4c0 2024-03-29T11:35:33.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.486+0000 7f92209e7640 10 monclient(hunting): _negotiate 2024-03-29T11:35:33.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.486+0000 7f92209e7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.486+0000 7f92209e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.486+0000 7f92209e7640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.486+0000 7f92209e7640 1 -- 172.21.15.155:0/1500294489 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9208003520 con 0x7f921402b4c0 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 1 -- 172.21.15.155:0/1500294489 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198854013 0 0) 0x7f921402aba0 con 0x7f9214006890 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 10 monclient(hunting): _negotiate 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 1 -- 172.21.15.155:0/1500294489 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f921402afa0 con 0x7f9214006890 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 1 -- 172.21.15.155:0/1500294489 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3958468113 0 0) 0x7f921402afa0 con 0x7f9214006890 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 1 -- 172.21.15.155:0/1500294489 >> v1:172.21.15.183:6789/0 conn(0x7f921402b4c0 legacy=0x7f92140cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 1 -- 172.21.15.155:0/1500294489 >> v1:172.21.15.155:6789/0 conn(0x7f9214006c70 legacy=0x7f92140c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 10 monclient: _finish_hunting 0 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 1 monclient: found mon.noname-b 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:33.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 1 -- 172.21.15.155:0/1500294489 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921402ad30 con 0x7f9214006890 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 10 monclient: _finish_auth 0 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 10 monclient: _check_auth_tickets 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.487+0000 7f92209e7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.488+0000 7f92209e7640 1 -- 172.21.15.155:0/1500294489 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f921c04f730 con 0x7f9214006890 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.488+0000 7f92209e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.488+0000 7f92209e7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.488+0000 7f92209e7640 10 monclient: dump: 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:33.488 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.488+0000 7f92229eb640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.488+0000 7f92229eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.488+0000 7f92209e7640 1 -- 172.21.15.155:0/1500294489 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f921c0008d0 con 0x7f9214006890 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.488+0000 7f92209e7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.488+0000 7f92229eb640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.488+0000 7f92229eb640 4 set_mon_vals no callback set 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.489+0000 7f92209e7640 1 -- 172.21.15.155:0/1500294489 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f921c04f730 con 0x7f9214006890 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.489+0000 7f92209e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.489+0000 7f92209e7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.489+0000 7f92209e7640 10 monclient: dump: 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:33.489 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:33.490 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:33.490 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:33.490 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:33.490 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:33.490 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:33.490 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:33.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.489+0000 7f92229eb640 10 monclient: shutdown 2024-03-29T11:35:33.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.489+0000 7f92229eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:33.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.489+0000 7f92229eb640 1 -- 172.21.15.155:0/1500294489 >> v1:172.21.15.155:6790/0 conn(0x7f9214006890 legacy=0x7f92140c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:33.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.489+0000 7f92229eb640 1 -- 172.21.15.155:0/1500294489 shutdown_connections 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.489+0000 7f92229eb640 1 -- 172.21.15.155:0/1500294489 >> 172.21.15.155:0/1500294489 conn(0x7f92140058f0 msgr2=0x7f9214005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.489+0000 7f92229eb640 1 -- 172.21.15.155:0/1500294489 shutdown_connections 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.489+0000 7f92229eb640 1 -- 172.21.15.155:0/1500294489 wait complete. 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.490+0000 7f92229eb640 10 monclient: build_initial_monmap 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.490+0000 7f92229eb640 10 monclient: monmap: 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:35:33.492 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.490+0000 7f92229eb640 1 librados: starting msgr at 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.490+0000 7f92229eb640 1 librados: starting objecter 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 1 Processor -- start 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 1 -- start start 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 1 librados: setting wanted keys 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 1 librados: calling monclient init 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 10 monclient: init 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 10 monclient(hunting): picked mon.noname-c con 0x7f9214006890 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 10 monclient(hunting): picked mon.noname-a con 0x7f921402b4c0 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 10 monclient(hunting): picked mon.noname-b con 0x7f92140c1a20 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92140c3e10 con 0x7f921402b4c0 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92140c4080 con 0x7f92140c1a20 2024-03-29T11:35:33.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92140c1fd0 con 0x7f9214006890 2024-03-29T11:35:33.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.491+0000 7f92229eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:33.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.492+0000 7f92229eb640 10 monclient(hunting): authenticate will time out at 915.617879s 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.496+0000 7f92219e9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f9214006890 0x7f92140c27d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:59981/0 (socket says 172.21.15.155:59981) 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.496+0000 7f92219e9640 1 -- 172.21.15.155:0/460948358 learned_addr learned my addr 172.21.15.155:0/460948358 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.496+0000 7f92127fc640 10 client.?.objecter ms_handle_connect 0x7f9214006890 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.496+0000 7f92127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.496+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612420230 0 0) 0x7f92140c1fd0 con 0x7f9214006890 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.496+0000 7f92127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.496+0000 7f92127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.496+0000 7f92127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.496+0000 7f92127fc640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.496+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9200004eb0 con 0x7f9214006890 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.497+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2149012675 0 0) 0x7f9200004eb0 con 0x7f9214006890 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.497+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92140c1fd0 con 0x7f9214006890 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.497+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f921c04f730 con 0x7f9214006890 2024-03-29T11:35:33.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.497+0000 7f92127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.497+0000 7f92127fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.497+0000 7f92127fc640 10 monclient(hunting): dump: 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.497+0000 7f92127fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.498+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 634112473 0 0) 0x7f92140c1fd0 con 0x7f9214006890 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.498+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 >> v1:172.21.15.183:6789/0 conn(0x7f92140c1a20 legacy=0x7f92141587b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:33.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.498+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 >> v1:172.21.15.155:6789/0 conn(0x7f921402b4c0 legacy=0x7f92140b9de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:33.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.499+0000 7f92127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:33.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.499+0000 7f92127fc640 1 monclient: found mon.c 2024-03-29T11:35:33.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.499+0000 7f92127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:33.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.499+0000 7f92127fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:35:33.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.499+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92140ba4f0 con 0x7f9214006890 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.499+0000 7f92127fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.499+0000 7f92127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.499+0000 7f92127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.500+0000 7f92229eb640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.500+0000 7f92229eb640 10 monclient: _renew_subs 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.500+0000 7f92229eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.500+0000 7f92229eb640 1 -- 172.21.15.155:0/460948358 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92140ba6e0 con 0x7f9214006890 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.500+0000 7f92229eb640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.500+0000 7f92229eb640 10 monclient: _renew_subs 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.500+0000 7f92229eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.500+0000 7f92229eb640 1 -- 172.21.15.155:0/460948358 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92141590a0 con 0x7f9214006890 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.500+0000 7f92229eb640 1 librados: init done 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.501+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f921c04fdd0 con 0x7f9214006890 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.501+0000 7f92127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:33.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.501+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f921c067d90 con 0x7f9214006890 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.501+0000 7f92127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.501+0000 7f92127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.501+0000 7f92127fc640 10 monclient: dump: 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.501+0000 7f92137fe640 4 set_mon_vals no callback set 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.501+0000 7f92229eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:33.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.502+0000 7f92229eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:33.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.503+0000 7f92229eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:35:33.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.503+0000 7f92229eb640 1 -- 172.21.15.155:0/460948358 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92140c2160 con 0x7f9214006890 2024-03-29T11:35:33.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.504+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f921c067ef0 con 0x7f9214006890 2024-03-29T11:35:33.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.505+0000 7f92211e8640 10 monclient: get_auth_request con 0x7f920006db50 auth_method 0 2024-03-29T11:35:33.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.505+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f921c0cf440 con 0x7f9214006890 2024-03-29T11:35:33.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.505+0000 7f92127fc640 10 client.4265.objecter ms_dispatch 0x7f921c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:33.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.505+0000 7f92127fc640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:33.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.505+0000 7f92127fc640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:33.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.505+0000 7f92127fc640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T11:35:33.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.505+0000 7f92127fc640 10 client.4265.objecter ms_handle_connect 0x7f920006db50 2024-03-29T11:35:33.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.513+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f921c0cb6e0 con 0x7f9214006890 2024-03-29T11:35:33.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.513+0000 7f92127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:33.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.513+0000 7f92127fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:33.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.838+0000 7f92229eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:33.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.838+0000 7f92229eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:33.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.838+0000 7f92229eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:35:33.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.838+0000 7f92229eb640 1 -- 172.21.15.155:0/460948358 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92140ba870 con 0x7f9214006890 2024-03-29T11:35:33.840 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:35:33.840 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:33.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.841+0000 7f92127fc640 1 -- 172.21.15.155:0/460948358 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f921c0cb6e0 con 0x7f9214006890 2024-03-29T11:35:33.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.841+0000 7f92127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:33.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.841+0000 7f92127fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:33.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.845+0000 7f92229eb640 10 librados: watch_flush enter 2024-03-29T11:35:33.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.848+0000 7f92229eb640 10 librados: watch_flush exit 2024-03-29T11:35:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.848+0000 7f92229eb640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T11:35:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.848+0000 7f92229eb640 10 client.4265.objecter successfully canceled tick 2024-03-29T11:35:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.849+0000 7f92229eb640 1 -- 172.21.15.155:0/460948358 >> v1:172.21.15.155:6816/1760814708 conn(0x7f920006db50 legacy=0x7f9200070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.849+0000 7f92229eb640 10 monclient: shutdown 2024-03-29T11:35:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.849+0000 7f92229eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.849+0000 7f92229eb640 1 -- 172.21.15.155:0/460948358 >> v1:172.21.15.155:6790/0 conn(0x7f9214006890 legacy=0x7f92140c27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.849+0000 7f92229eb640 1 -- 172.21.15.155:0/460948358 shutdown_connections 2024-03-29T11:35:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.849+0000 7f92229eb640 1 -- 172.21.15.155:0/460948358 >> 172.21.15.155:0/460948358 conn(0x7f92140058f0 msgr2=0x7f9214038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.849+0000 7f92229eb640 1 -- 172.21.15.155:0/460948358 shutdown_connections 2024-03-29T11:35:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.849+0000 7f92229eb640 1 -- 172.21.15.155:0/460948358 wait complete. 2024-03-29T11:35:33.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:33.850+0000 7f92229eb640 1 librados: shutdown 2024-03-29T11:35:33.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:40.671 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:40.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.837+0000 7f2c5cadd640 10 monclient: get_monmap_and_config 2024-03-29T11:35:40.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.837+0000 7f2c5cadd640 10 monclient: build_initial_monmap 2024-03-29T11:35:40.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.837+0000 7f2c5cadd640 10 monclient: monmap: 2024-03-29T11:35:40.837 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:40.837 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:40.837 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:35:40.837642+0000 2024-03-29T11:35:40.837 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:35:40.837642+0000 2024-03-29T11:35:40.837 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:40.837 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:40.837 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:40.837 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.838+0000 7f2c5cadd640 1 Processor -- start 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.838+0000 7f2c5cadd640 1 -- start start 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.838+0000 7f2c5cadd640 10 monclient: init 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.839+0000 7f2c5cadd640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.839+0000 7f2c5cadd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.839+0000 7f2c5cadd640 10 monclient(hunting): picked mon.noname-a con 0x7f2c580a9930 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.839+0000 7f2c5cadd640 10 monclient(hunting): picked mon.noname-b con 0x7f2c58149dc0 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.839+0000 7f2c5cadd640 10 monclient(hunting): picked mon.noname-c con 0x7f2c5814ff40 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.839+0000 7f2c5cadd640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c580aae60 con 0x7f2c580a9930 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.839+0000 7f2c5cadd640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c580b4310 con 0x7f2c5814ff40 2024-03-29T11:35:40.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.839+0000 7f2c5cadd640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c580afdd0 con 0x7f2c58149dc0 2024-03-29T11:35:40.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.839+0000 7f2c5cadd640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:40.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.839+0000 7f2c5cadd640 10 monclient(hunting): authenticate will time out at 922.965572s 2024-03-29T11:35:40.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.847+0000 7f2c5659c640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f2c580a9930 0x7f2c58143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:47555/0 (socket says 172.21.15.155:47555) 2024-03-29T11:35:40.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.847+0000 7f2c5659c640 1 -- 172.21.15.155:0/3638664933 learned_addr learned my addr 172.21.15.155:0/3638664933 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:40.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 1 -- 172.21.15.155:0/3638664933 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563869911 0 0) 0x7f2c580aae60 con 0x7f2c580a9930 2024-03-29T11:35:40.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:35:40.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 1 -- 172.21.15.155:0/3638664933 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c40003520 con 0x7f2c580a9930 2024-03-29T11:35:40.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 1 -- 172.21.15.155:0/3638664933 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223131962 0 0) 0x7f2c580afdd0 con 0x7f2c58149dc0 2024-03-29T11:35:40.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:35:40.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.848+0000 7f2c5559a640 1 -- 172.21.15.155:0/3638664933 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c580aae60 con 0x7f2c58149dc0 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 1 -- 172.21.15.155:0/3638664933 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1882373457 0 0) 0x7f2c580aae60 con 0x7f2c58149dc0 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 1 -- 172.21.15.155:0/3638664933 >> v1:172.21.15.183:6789/0 conn(0x7f2c5814ff40 legacy=0x7f2c58152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 1 -- 172.21.15.155:0/3638664933 >> v1:172.21.15.155:6789/0 conn(0x7f2c580a9930 legacy=0x7f2c58143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 1 monclient: found mon.noname-b 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 1 -- 172.21.15.155:0/3638664933 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5814a950 con 0x7f2c58149dc0 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 10 monclient: _finish_auth 0 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:40.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 1 -- 172.21.15.155:0/3638664933 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2c480015a0 con 0x7f2c58149dc0 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 10 monclient: dump: 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:40.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 1 -- 172.21.15.155:0/3638664933 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c48001700 con 0x7f2c58149dc0 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 1 -- 172.21.15.155:0/3638664933 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2c48001b50 con 0x7f2c58149dc0 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.850+0000 7f2c5559a640 10 monclient: dump: 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:40.854 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:40.855 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:40.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.853+0000 7f2c5cadd640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:35:40.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.853+0000 7f2c5cadd640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:40.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.853+0000 7f2c5cadd640 4 set_mon_vals no callback set 2024-03-29T11:35:40.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.854+0000 7f2c5cadd640 10 monclient: shutdown 2024-03-29T11:35:40.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.854+0000 7f2c5cadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:40.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.854+0000 7f2c5cadd640 1 -- 172.21.15.155:0/3638664933 >> v1:172.21.15.155:6790/0 conn(0x7f2c58149dc0 legacy=0x7f2c5814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.854+0000 7f2c5cadd640 1 -- 172.21.15.155:0/3638664933 shutdown_connections 2024-03-29T11:35:40.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.854+0000 7f2c5cadd640 1 -- 172.21.15.155:0/3638664933 >> 172.21.15.155:0/3638664933 conn(0x7f2c580b35e0 msgr2=0x7f2c580b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:40.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.855+0000 7f2c5cadd640 1 -- 172.21.15.155:0/3638664933 shutdown_connections 2024-03-29T11:35:40.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.855+0000 7f2c5cadd640 1 -- 172.21.15.155:0/3638664933 wait complete. 2024-03-29T11:35:40.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.856+0000 7f2c5cadd640 10 monclient: build_initial_monmap 2024-03-29T11:35:40.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.856+0000 7f2c5cadd640 10 monclient: monmap: 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.856+0000 7f2c5cadd640 1 librados: starting msgr at 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.856+0000 7f2c5cadd640 1 librados: starting objecter 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 1 Processor -- start 2024-03-29T11:35:40.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 1 -- start start 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 1 librados: setting wanted keys 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 1 librados: calling monclient init 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 10 monclient: init 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 10 monclient(hunting): picked mon.noname-c con 0x7f2c580a9930 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 10 monclient(hunting): picked mon.noname-a con 0x7f2c58149dc0 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 10 monclient(hunting): picked mon.noname-b con 0x7f2c5814ff40 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c58146040 con 0x7f2c58149dc0 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c58146f80 con 0x7f2c5814ff40 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c58147260 con 0x7f2c580a9930 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:40.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.857+0000 7f2c5cadd640 10 monclient(hunting): authenticate will time out at 922.983866s 2024-03-29T11:35:40.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.864+0000 7f2c56d9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f2c5814ff40 0x7f2c582000a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:56861/0 (socket says 172.21.15.155:56861) 2024-03-29T11:35:40.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.864+0000 7f2c56d9d640 1 -- 172.21.15.155:0/2178388981 learned_addr learned my addr 172.21.15.155:0/2178388981 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:40.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.866+0000 7f2c46ffd640 10 client.?.objecter ms_handle_connect 0x7f2c58149dc0 2024-03-29T11:35:40.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.866+0000 7f2c46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:40.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.866+0000 7f2c46ffd640 10 client.?.objecter ms_handle_connect 0x7f2c5814ff40 2024-03-29T11:35:40.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.866+0000 7f2c46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:40.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.869+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896885833 0 0) 0x7f2c58146f80 con 0x7f2c5814ff40 2024-03-29T11:35:40.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.869+0000 7f2c46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.869+0000 7f2c46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.869+0000 7f2c46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.869+0000 7f2c46ffd640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:35:40.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.869+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c38004eb0 con 0x7f2c5814ff40 2024-03-29T11:35:40.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.870+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197787568 0 0) 0x7f2c58146040 con 0x7f2c58149dc0 2024-03-29T11:35:40.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.870+0000 7f2c46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.870+0000 7f2c46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.870+0000 7f2c46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.870+0000 7f2c46ffd640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.870+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c58146f80 con 0x7f2c58149dc0 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.871+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4024675439 0 0) 0x7f2c58146f80 con 0x7f2c58149dc0 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.871+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c58146040 con 0x7f2c58149dc0 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.871+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2c48001af0 con 0x7f2c58149dc0 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.871+0000 7f2c46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.871+0000 7f2c46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.871+0000 7f2c46ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:40.870 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.871 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:40.871 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:40.871 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:40.871 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:40.871 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:40.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.871+0000 7f2c46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:40.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.871+0000 7f2c46ffd640 10 client.?.objecter ms_handle_connect 0x7f2c580a9930 2024-03-29T11:35:40.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.871+0000 7f2c46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:40.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.871+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2659063654 0 0) 0x7f2c38004eb0 con 0x7f2c5814ff40 2024-03-29T11:35:40.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.871+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c58146f80 con 0x7f2c5814ff40 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2c4c0018b0 con 0x7f2c5814ff40 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:40.874 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3058793576 0 0) 0x7f2c58146040 con 0x7f2c58149dc0 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 >> v1:172.21.15.155:6790/0 conn(0x7f2c580a9930 legacy=0x7f2c58145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 >> v1:172.21.15.183:6789/0 conn(0x7f2c5814ff40 legacy=0x7f2c582000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 1 monclient: found mon.a 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c58147540 con 0x7f2c58149dc0 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c5cadd640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c5cadd640 10 monclient: _renew_subs 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c5cadd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:40.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c5cadd640 1 -- 172.21.15.155:0/2178388981 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5813cb00 con 0x7f2c58149dc0 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c5cadd640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c5cadd640 10 monclient: _renew_subs 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c5cadd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c5cadd640 1 -- 172.21.15.155:0/2178388981 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c581453a0 con 0x7f2c58149dc0 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c5cadd640 1 librados: init done 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c480038a0 con 0x7f2c58149dc0 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2c48003bb0 con 0x7f2c58149dc0 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c46ffd640 10 monclient: dump: 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:40.876 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.872+0000 7f2c54d99640 4 set_mon_vals no callback set 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.874+0000 7f2c5cadd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.874+0000 7f2c5cadd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.874+0000 7f2c5cadd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:40.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.874+0000 7f2c5cadd640 1 -- 172.21.15.155:0/2178388981 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3c005080 con 0x7f2c58149dc0 2024-03-29T11:35:40.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.876+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f2c48019b80 con 0x7f2c58149dc0 2024-03-29T11:35:40.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.877+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f2c480839b0 con 0x7f2c58149dc0 2024-03-29T11:35:40.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.877+0000 7f2c46ffd640 10 client.4269.objecter ms_dispatch 0x7f2c58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:40.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.877+0000 7f2c46ffd640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:40.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.877+0000 7f2c46ffd640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:40.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.877+0000 7f2c46ffd640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-29T11:35:40.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.877+0000 7f2c5659c640 10 monclient: get_auth_request con 0x7f2c3806de10 auth_method 0 2024-03-29T11:35:40.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.878+0000 7f2c46ffd640 10 client.4269.objecter ms_handle_connect 0x7f2c3806de10 2024-03-29T11:35:40.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.888+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c4804f290 con 0x7f2c58149dc0 2024-03-29T11:35:40.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.888+0000 7f2c46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:40.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:40.888+0000 7f2c46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:41.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.265+0000 7f2c5cadd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:41.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.266+0000 7f2c5cadd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:41.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.266+0000 7f2c5cadd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:41.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.266+0000 7f2c5cadd640 1 -- 172.21.15.155:0/2178388981 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c3c005330 con 0x7f2c58149dc0 2024-03-29T11:35:41.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.267+0000 7f2c46ffd640 1 -- 172.21.15.155:0/2178388981 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f2c480511c0 con 0x7f2c58149dc0 2024-03-29T11:35:41.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.267+0000 7f2c46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:41.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.267+0000 7f2c46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:41.269 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:35:41.269 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:41.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.272+0000 7f2c5cadd640 10 librados: watch_flush enter 2024-03-29T11:35:41.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.272+0000 7f2c5cadd640 10 librados: watch_flush exit 2024-03-29T11:35:41.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.272+0000 7f2c5cadd640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-29T11:35:41.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.272+0000 7f2c5cadd640 10 client.4269.objecter successfully canceled tick 2024-03-29T11:35:41.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.274+0000 7f2c5cadd640 1 -- 172.21.15.155:0/2178388981 >> v1:172.21.15.155:6816/1760814708 conn(0x7f2c3806de10 legacy=0x7f2c380702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:41.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.274+0000 7f2c5cadd640 10 monclient: shutdown 2024-03-29T11:35:41.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.274+0000 7f2c5cadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:41.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.274+0000 7f2c5cadd640 1 -- 172.21.15.155:0/2178388981 >> v1:172.21.15.155:6789/0 conn(0x7f2c58149dc0 legacy=0x7f2c5813c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:41.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.275+0000 7f2c5cadd640 1 -- 172.21.15.155:0/2178388981 shutdown_connections 2024-03-29T11:35:41.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.275+0000 7f2c5cadd640 1 -- 172.21.15.155:0/2178388981 >> 172.21.15.155:0/2178388981 conn(0x7f2c580b35e0 msgr2=0x7f2c580b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:41.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.275+0000 7f2c5cadd640 1 -- 172.21.15.155:0/2178388981 shutdown_connections 2024-03-29T11:35:41.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.275+0000 7f2c5cadd640 1 -- 172.21.15.155:0/2178388981 wait complete. 2024-03-29T11:35:41.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:41.275+0000 7f2c5cadd640 1 librados: shutdown 2024-03-29T11:35:41.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:48.097 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:48.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.261+0000 7fccc359e640 10 monclient: get_monmap_and_config 2024-03-29T11:35:48.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.261+0000 7fccc359e640 10 monclient: build_initial_monmap 2024-03-29T11:35:48.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.261+0000 7fccc359e640 10 monclient: monmap: 2024-03-29T11:35:48.260 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:48.261 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:48.261 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:35:48.263062+0000 2024-03-29T11:35:48.261 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:35:48.263062+0000 2024-03-29T11:35:48.261 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:48.261 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:48.261 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:48.261 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:35:48.261 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:35:48.261 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:48.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.262+0000 7fccc359e640 1 Processor -- start 2024-03-29T11:35:48.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.263+0000 7fccc359e640 1 -- start start 2024-03-29T11:35:48.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.263+0000 7fccc359e640 10 monclient: init 2024-03-29T11:35:48.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.264+0000 7fccc359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:48.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.264+0000 7fccc359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:48.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.264+0000 7fccc359e640 10 monclient(hunting): picked mon.noname-c con 0x7fccbc038c60 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:48.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.264+0000 7fccc359e640 10 monclient(hunting): picked mon.noname-a con 0x7fccbc0324b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:48.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.264+0000 7fccc359e640 10 monclient(hunting): picked mon.noname-b con 0x7fccbc0d11e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:48.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.264+0000 7fccc359e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccbc0caf80 con 0x7fccbc0324b0 2024-03-29T11:35:48.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.264+0000 7fccc359e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccbc039850 con 0x7fccbc038c60 2024-03-29T11:35:48.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.264+0000 7fccc359e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccbc0c78d0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.264+0000 7fccc359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:48.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.264+0000 7fccc359e640 10 monclient(hunting): authenticate will time out at 930.391816s 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.265+0000 7fccc259c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fccbc038c60 0x7fccbc0c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:60550/0 (socket says 172.21.15.155:60550) 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.265+0000 7fccc259c640 1 -- 172.21.15.155:0/57604999 learned_addr learned my addr 172.21.15.155:0/57604999 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.265+0000 7fccc159a640 1 -- 172.21.15.155:0/57604999 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076305708 0 0) 0x7fccbc039850 con 0x7fccbc038c60 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.265+0000 7fccc159a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.265+0000 7fccc159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.265+0000 7fccc159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.265+0000 7fccc159a640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.265+0000 7fccc159a640 1 -- 172.21.15.155:0/57604999 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccac003520 con 0x7fccbc038c60 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 1 -- 172.21.15.155:0/57604999 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723965993 0 0) 0x7fccbc0c78d0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 1 -- 172.21.15.155:0/57604999 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccbc039850 con 0x7fccbc0d11e0 2024-03-29T11:35:48.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 1 -- 172.21.15.155:0/57604999 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 691633554 0 0) 0x7fccac003520 con 0x7fccbc038c60 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 1 -- 172.21.15.155:0/57604999 >> v1:172.21.15.155:6790/0 conn(0x7fccbc0d11e0 legacy=0x7fccbc0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 1 -- 172.21.15.155:0/57604999 >> v1:172.21.15.155:6789/0 conn(0x7fccbc0324b0 legacy=0x7fccbc0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 1 monclient: found mon.noname-c 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 1 -- 172.21.15.155:0/57604999 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc0c7a60 con 0x7fccbc038c60 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient: _finish_auth 0 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 1 -- 172.21.15.155:0/57604999 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fccb4001690 con 0x7fccbc038c60 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient: dump: 2024-03-29T11:35:48.268 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:48.269 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:48.269 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:48.269 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:48.269 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:48.269 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:48.269 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:48.269 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:48.269 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:48.269 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:48.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc359e640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:35:48.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:48.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 1 -- 172.21.15.155:0/57604999 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccb4001860 con 0x7fccbc038c60 2024-03-29T11:35:48.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:48.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 1 -- 172.21.15.155:0/57604999 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fccb4001c70 con 0x7fccbc038c60 2024-03-29T11:35:48.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.266+0000 7fccc159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:48.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.267+0000 7fccc159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:48.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.267+0000 7fccc159a640 10 monclient: dump: 2024-03-29T11:35:48.275 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:48.275 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:48.275 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.267+0000 7fccc359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.267+0000 7fccc359e640 4 set_mon_vals no callback set 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.267+0000 7fccc359e640 10 monclient: shutdown 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.267+0000 7fccc359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:48.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.267+0000 7fccc359e640 1 -- 172.21.15.155:0/57604999 >> v1:172.21.15.183:6789/0 conn(0x7fccbc038c60 legacy=0x7fccbc0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.268+0000 7fccc359e640 1 -- 172.21.15.155:0/57604999 shutdown_connections 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.268+0000 7fccc359e640 1 -- 172.21.15.155:0/57604999 >> 172.21.15.155:0/57604999 conn(0x7fccbc005c20 msgr2=0x7fccbc006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.268+0000 7fccc359e640 1 -- 172.21.15.155:0/57604999 shutdown_connections 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.268+0000 7fccc359e640 1 -- 172.21.15.155:0/57604999 wait complete. 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.269+0000 7fccc359e640 10 monclient: build_initial_monmap 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.269+0000 7fccc359e640 10 monclient: monmap: 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:48.277 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.270+0000 7fccc359e640 1 librados: starting msgr at 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.270+0000 7fccc359e640 1 librados: starting objecter 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.270+0000 7fccc359e640 1 Processor -- start 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.270+0000 7fccc359e640 1 -- start start 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.270+0000 7fccc359e640 1 librados: setting wanted keys 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.270+0000 7fccc359e640 1 librados: calling monclient init 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.270+0000 7fccc359e640 10 monclient: init 2024-03-29T11:35:48.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc359e640 10 monclient(hunting): picked mon.noname-c con 0x7fccbc0324b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc359e640 10 monclient(hunting): picked mon.noname-a con 0x7fccbc038c60 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc359e640 10 monclient(hunting): picked mon.noname-b con 0x7fccbc0d11e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc359e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccbc02d170 con 0x7fccbc038c60 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc359e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccbc0097d0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc359e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccbc009ab0 con 0x7fccbc0324b0 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc359e640 10 monclient(hunting): authenticate will time out at 930.398349s 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc1d9b640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fccbc038c60 0x7fccbc1699f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36890/0 (socket says 172.21.15.155:36890) 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccc1d9b640 1 -- 172.21.15.155:0/14109782 learned_addr learned my addr 172.21.15.155:0/14109782 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccb2ffd640 10 client.?.objecter ms_handle_connect 0x7fccbc038c60 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccb2ffd640 10 client.?.objecter ms_handle_connect 0x7fccbc0d11e0 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.271+0000 7fccb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364991600 0 0) 0x7fccbc0097d0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca4004eb0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184155069 0 0) 0x7fccbc02d170 con 0x7fccbc038c60 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:48.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:48.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:35:48.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccbc0097d0 con 0x7fccbc038c60 2024-03-29T11:35:48.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3661951146 0 0) 0x7fcca4004eb0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccbc02d170 con 0x7fccbc0d11e0 2024-03-29T11:35:48.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fccc404fdd0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:48.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:48.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:48.281 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:48.281 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:48.282 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 client.?.objecter ms_handle_connect 0x7fccbc0324b0 2024-03-29T11:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.272+0000 7fccb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2806928314 0 0) 0x7fccbc0097d0 con 0x7fccbc038c60 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcca4004eb0 con 0x7fccbc038c60 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fccb8001a30 con 0x7fccbc038c60 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 486514685 0 0) 0x7fccbc02d170 con 0x7fccbc0d11e0 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 >> v1:172.21.15.155:6790/0 conn(0x7fccbc0324b0 legacy=0x7fccbc02ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 >> v1:172.21.15.155:6789/0 conn(0x7fccbc038c60 legacy=0x7fccbc1699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 1 monclient: found mon.b 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc036830 con 0x7fccbc0d11e0 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccc359e640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:35:48.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccc359e640 10 monclient: _renew_subs 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccc359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccc359e640 1 -- 172.21.15.155:0/14109782 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccbc036ae0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccc359e640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccc359e640 10 monclient: _renew_subs 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccc359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccc359e640 1 -- 172.21.15.155:0/14109782 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccbc16a2e0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.273+0000 7fccc359e640 1 librados: init done 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.274+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccc404f400 con 0x7fccbc0d11e0 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.274+0000 7fccb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.274+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fccc4075710 con 0x7fccbc0d11e0 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.274+0000 7fccb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.274+0000 7fccb2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.274+0000 7fccb2ffd640 10 monclient: dump: 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:48.285 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.274+0000 7fccc0d99640 4 set_mon_vals no callback set 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.275+0000 7fccc359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.276+0000 7fccc359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.276+0000 7fccc359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:35:48.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.276+0000 7fccc359e640 1 -- 172.21.15.155:0/14109782 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca8005080 con 0x7fccbc0d11e0 2024-03-29T11:35:48.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.277+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fccc4067bf0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.278+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fccc40fe2a0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.278+0000 7fccb2ffd640 10 client.4276.objecter ms_dispatch 0x7fccc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:48.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.278+0000 7fccb2ffd640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:48.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.278+0000 7fccb2ffd640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:48.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.278+0000 7fccc259c640 10 monclient: get_auth_request con 0x7fcca406dc90 auth_method 0 2024-03-29T11:35:48.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.278+0000 7fccb2ffd640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-29T11:35:48.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.278+0000 7fccb2ffd640 10 client.4276.objecter ms_handle_connect 0x7fcca406dc90 2024-03-29T11:35:48.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.282+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fccc40c9b80 con 0x7fccbc0d11e0 2024-03-29T11:35:48.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.282+0000 7fccb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:48.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.282+0000 7fccb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:48.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.667+0000 7fccc359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:48.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.667+0000 7fccc359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:48.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.667+0000 7fccc359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:35:48.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.667+0000 7fccc359e640 1 -- 172.21.15.155:0/14109782 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcca8005330 con 0x7fccbc0d11e0 2024-03-29T11:35:48.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.668+0000 7fccb2ffd640 1 -- 172.21.15.155:0/14109782 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fccc40cbab0 con 0x7fccbc0d11e0 2024-03-29T11:35:48.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.668+0000 7fccb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:48.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.668+0000 7fccb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:48.667 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:35:48.667 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:48.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.671+0000 7fccc359e640 10 librados: watch_flush enter 2024-03-29T11:35:48.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 10 librados: watch_flush exit 2024-03-29T11:35:48.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-29T11:35:48.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 10 client.4276.objecter successfully canceled tick 2024-03-29T11:35:48.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 1 -- 172.21.15.155:0/14109782 >> v1:172.21.15.155:6816/1760814708 conn(0x7fcca406dc90 legacy=0x7fcca4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:48.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 10 monclient: shutdown 2024-03-29T11:35:48.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:48.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 1 -- 172.21.15.155:0/14109782 >> v1:172.21.15.183:6789/0 conn(0x7fccbc0d11e0 legacy=0x7fccbc174130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:48.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 1 -- 172.21.15.155:0/14109782 shutdown_connections 2024-03-29T11:35:48.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 1 -- 172.21.15.155:0/14109782 >> 172.21.15.155:0/14109782 conn(0x7fccbc005c20 msgr2=0x7fccbc0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:48.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 1 -- 172.21.15.155:0/14109782 shutdown_connections 2024-03-29T11:35:48.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 1 -- 172.21.15.155:0/14109782 wait complete. 2024-03-29T11:35:48.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:48.672+0000 7fccc359e640 1 librados: shutdown 2024-03-29T11:35:48.686 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:55.494 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.648+0000 7f0bc04c1640 10 monclient: get_monmap_and_config 2024-03-29T11:35:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.648+0000 7f0bc04c1640 10 monclient: build_initial_monmap 2024-03-29T11:35:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.648+0000 7f0bc04c1640 10 monclient: monmap: 2024-03-29T11:35:55.649 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:55.649 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:55.649 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:35:55.650334+0000 2024-03-29T11:35:55.649 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:35:55.650334+0000 2024-03-29T11:35:55.649 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:55.649 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:55.649 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:55.649 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:35:55.649 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:35:55.649 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:55.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.650+0000 7f0bc04c1640 1 Processor -- start 2024-03-29T11:35:55.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.650+0000 7f0bc04c1640 1 -- start start 2024-03-29T11:35:55.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.650+0000 7f0bc04c1640 10 monclient: init 2024-03-29T11:35:55.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.651+0000 7f0bc04c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:55.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.651+0000 7f0bc04c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:55.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.652+0000 7f0bc04c1640 10 monclient(hunting): picked mon.noname-a con 0x7f0bb80ae250 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:55.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.652+0000 7f0bc04c1640 10 monclient(hunting): picked mon.noname-b con 0x7f0bb813f970 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:55.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.652+0000 7f0bc04c1640 10 monclient(hunting): picked mon.noname-c con 0x7f0bb8142800 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:55.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.652+0000 7f0bc04c1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb80afe00 con 0x7f0bb80ae250 2024-03-29T11:35:55.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.652+0000 7f0bc04c1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb8142c10 con 0x7f0bb8142800 2024-03-29T11:35:55.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.652+0000 7f0bc04c1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb80afb20 con 0x7f0bb813f970 2024-03-29T11:35:55.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.652+0000 7f0bc04c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:55.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.652+0000 7f0bc04c1640 10 monclient(hunting): authenticate will time out at 937.779522s 2024-03-29T11:35:55.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.654+0000 7f0bbe25d640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f0bb80ae250 0x7f0bb813f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:43023/0 (socket says 172.21.15.155:43023) 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.654+0000 7f0bbe25d640 1 -- 172.21.15.155:0/1824143645 learned_addr learned my addr 172.21.15.155:0/1824143645 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.654+0000 7f0bbd25b640 1 -- 172.21.15.155:0/1824143645 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943302334 0 0) 0x7f0bb80afe00 con 0x7f0bb80ae250 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.654+0000 7f0bbd25b640 10 monclient(hunting): _negotiate 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.654+0000 7f0bbd25b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.654+0000 7f0bbd25b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.654+0000 7f0bbd25b640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.654+0000 7f0bbd25b640 1 -- 172.21.15.155:0/1824143645 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba4003520 con 0x7f0bb80ae250 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 1 -- 172.21.15.155:0/1824143645 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4166178850 0 0) 0x7f0ba4003520 con 0x7f0bb80ae250 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 1 -- 172.21.15.155:0/1824143645 >> v1:172.21.15.155:6790/0 conn(0x7f0bb813f970 legacy=0x7f0bb81420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 1 -- 172.21.15.155:0/1824143645 >> v1:172.21.15.183:6789/0 conn(0x7f0bb8142800 legacy=0x7f0bb814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 10 monclient: _finish_hunting 0 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 1 monclient: found mon.noname-a 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:35:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 1 -- 172.21.15.155:0/1824143645 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb80aab80 con 0x7f0bb80ae250 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 10 monclient: _finish_auth 0 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 10 monclient: _check_auth_tickets 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 1 -- 172.21.15.155:0/1824143645 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0bb4001690 con 0x7f0bb80ae250 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 10 monclient: dump: 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:55.656 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:55.657 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:55.657 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:55.657 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:55.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 1 -- 172.21.15.155:0/1824143645 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bb40018f0 con 0x7f0bb80ae250 2024-03-29T11:35:55.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:55.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 1 -- 172.21.15.155:0/1824143645 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0bb4001d40 con 0x7f0bb80ae250 2024-03-29T11:35:55.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:55.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:55.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.655+0000 7f0bbd25b640 10 monclient: dump: 2024-03-29T11:35:55.658 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:55.658 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:55.658 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:55.658 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.657+0000 7f0bc04c1640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.657+0000 7f0bc04c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.657+0000 7f0bc04c1640 4 set_mon_vals no callback set 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.658+0000 7f0bc04c1640 10 monclient: shutdown 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.658+0000 7f0bc04c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:55.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.658+0000 7f0bc04c1640 1 -- 172.21.15.155:0/1824143645 >> v1:172.21.15.155:6789/0 conn(0x7f0bb80ae250 legacy=0x7f0bb813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.659+0000 7f0bc04c1640 1 -- 172.21.15.155:0/1824143645 shutdown_connections 2024-03-29T11:35:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.659+0000 7f0bc04c1640 1 -- 172.21.15.155:0/1824143645 >> 172.21.15.155:0/1824143645 conn(0x7f0bb80b35e0 msgr2=0x7f0bb80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.659+0000 7f0bc04c1640 1 -- 172.21.15.155:0/1824143645 shutdown_connections 2024-03-29T11:35:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.659+0000 7f0bc04c1640 1 -- 172.21.15.155:0/1824143645 wait complete. 2024-03-29T11:35:55.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.660+0000 7f0bc04c1640 10 monclient: build_initial_monmap 2024-03-29T11:35:55.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.660+0000 7f0bc04c1640 10 monclient: monmap: 2024-03-29T11:35:55.662 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:35:55.662 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:55.662 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:35:55.662 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:35:55.662 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:55.662 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:55.662 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:35:55.662 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:35:55.662 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.660+0000 7f0bc04c1640 1 librados: starting msgr at 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.660+0000 7f0bc04c1640 1 librados: starting objecter 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.660+0000 7f0bc04c1640 1 Processor -- start 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 1 -- start start 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 1 librados: setting wanted keys 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 1 librados: calling monclient init 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 10 monclient: init 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 10 monclient(hunting): picked mon.noname-b con 0x7f0bb80ae250 addr v1:172.21.15.183:6789/0 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 10 monclient(hunting): picked mon.noname-c con 0x7f0bb813f970 addr v1:172.21.15.155:6790/0 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 10 monclient(hunting): picked mon.noname-a con 0x7f0bb8142800 addr v1:172.21.15.155:6789/0 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb8144290 con 0x7f0bb8142800 2024-03-29T11:35:55.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb81458a0 con 0x7f0bb80ae250 2024-03-29T11:35:55.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb8145b80 con 0x7f0bb813f970 2024-03-29T11:35:55.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:55.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.661+0000 7f0bc04c1640 10 monclient(hunting): authenticate will time out at 937.788821s 2024-03-29T11:35:55.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.665+0000 7f0bbda5c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f0bb813f970 0x7f0bb80ab0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:46551/0 (socket says 172.21.15.155:46551) 2024-03-29T11:35:55.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.665+0000 7f0bbda5c640 1 -- 172.21.15.155:0/2312013935 learned_addr learned my addr 172.21.15.155:0/2312013935 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:35:55.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.665+0000 7f0baeffd640 10 client.?.objecter ms_handle_connect 0x7f0bb813f970 2024-03-29T11:35:55.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.665+0000 7f0baeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:55.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.666+0000 7f0baeffd640 10 client.?.objecter ms_handle_connect 0x7f0bb80ae250 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.666+0000 7f0baeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.666+0000 7f0baeffd640 10 client.?.objecter ms_handle_connect 0x7f0bb8142800 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.666+0000 7f0baeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.666+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835906825 0 0) 0x7f0bb81458a0 con 0x7f0bb80ae250 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.666+0000 7f0baeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.666+0000 7f0baeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.666+0000 7f0baeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.666+0000 7f0baeffd640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.666+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b9c004eb0 con 0x7f0bb80ae250 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231176804 0 0) 0x7f0bb8144290 con 0x7f0bb8142800 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb81458a0 con 0x7f0bb8142800 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3746496499 0 0) 0x7f0bb81458a0 con 0x7f0bb8142800 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bb8144290 con 0x7f0bb8142800 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0bb0001a70 con 0x7f0bb8142800 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): dump: 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:55.668 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:55.669 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:55.669 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:55.669 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:55.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:55.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669522302 0 0) 0x7f0bb8145b80 con 0x7f0bb813f970 2024-03-29T11:35:55.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:55.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:55.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:55.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:35:55.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb81458a0 con 0x7f0bb813f970 2024-03-29T11:35:55.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2667949227 0 0) 0x7f0bb8144290 con 0x7f0bb8142800 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 >> v1:172.21.15.155:6790/0 conn(0x7f0bb813f970 legacy=0x7f0bb80ab0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 >> v1:172.21.15.183:6789/0 conn(0x7f0bb80ae250 legacy=0x7f0bb81dea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 monclient: found mon.a 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb8144990 con 0x7f0bb8142800 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.667+0000 7f0baeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.668+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bb0001bd0 con 0x7f0bb8142800 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.668+0000 7f0baeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.668+0000 7f0bbca5a640 4 set_mon_vals no callback set 2024-03-29T11:35:55.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0bb0003970 con 0x7f0bb8142800 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0baeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0baeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0baeffd640 10 monclient: dump: 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:35:55.673 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:35:55.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0bc04c1640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:35:55.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0bc04c1640 10 monclient: _renew_subs 2024-03-29T11:35:55.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0bc04c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:55.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0bc04c1640 1 -- 172.21.15.155:0/2312013935 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb8146380 con 0x7f0bb8142800 2024-03-29T11:35:55.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0bc04c1640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:55.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0bc04c1640 10 monclient: _renew_subs 2024-03-29T11:35:55.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0bc04c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:55.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0bc04c1640 1 -- 172.21.15.155:0/2312013935 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb8143700 con 0x7f0bb8142800 2024-03-29T11:35:55.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.669+0000 7f0bc04c1640 1 librados: init done 2024-03-29T11:35:55.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.670+0000 7f0bc04c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:55.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.670+0000 7f0bc04c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:55.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.670+0000 7f0bc04c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:55.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.670+0000 7f0bc04c1640 1 -- 172.21.15.155:0/2312013935 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb8144060 con 0x7f0bb8142800 2024-03-29T11:35:55.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.672+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f0bb0019940 con 0x7f0bb8142800 2024-03-29T11:35:55.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.673+0000 7f0bbe25d640 10 monclient: get_auth_request con 0x7f0b9c06dea0 auth_method 0 2024-03-29T11:35:55.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.673+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f0bb0083a10 con 0x7f0bb8142800 2024-03-29T11:35:55.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.673+0000 7f0baeffd640 10 client.4281.objecter ms_dispatch 0x7f0bb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:55.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.673+0000 7f0baeffd640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:55.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.673+0000 7f0baeffd640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:55.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.673+0000 7f0baeffd640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-29T11:35:55.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.673+0000 7f0baeffd640 10 client.4281.objecter ms_handle_connect 0x7f0b9c06dea0 2024-03-29T11:35:55.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.681+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0bb8144060 con 0x7f0bb8142800 2024-03-29T11:35:55.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.682+0000 7f0baeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:55.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:55.682+0000 7f0baeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:56.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.007+0000 7f0bc04c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.007+0000 7f0bc04c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.007+0000 7f0bc04c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:35:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.007+0000 7f0bc04c1640 1 -- 172.21.15.155:0/2312013935 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bb8144060 con 0x7f0bb8142800 2024-03-29T11:35:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.008+0000 7f0baeffd640 1 -- 172.21.15.155:0/2312013935 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f0bb8144060 con 0x7f0bb8142800 2024-03-29T11:35:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.008+0000 7f0baeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.008+0000 7f0baeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:56.007 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:35:56.008 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:56.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.012+0000 7f0bc04c1640 10 librados: watch_flush enter 2024-03-29T11:35:56.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.012+0000 7f0bc04c1640 10 librados: watch_flush exit 2024-03-29T11:35:56.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.012+0000 7f0bc04c1640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-29T11:35:56.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.012+0000 7f0bc04c1640 10 client.4281.objecter successfully canceled tick 2024-03-29T11:35:56.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.012+0000 7f0bc04c1640 1 -- 172.21.15.155:0/2312013935 >> v1:172.21.15.155:6816/1760814708 conn(0x7f0b9c06dea0 legacy=0x7f0b9c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.012+0000 7f0bc04c1640 10 monclient: shutdown 2024-03-29T11:35:56.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.012+0000 7f0bc04c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:56.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.012+0000 7f0bc04c1640 1 -- 172.21.15.155:0/2312013935 >> v1:172.21.15.155:6789/0 conn(0x7f0bb8142800 legacy=0x7f0bb81e5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:56.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.014+0000 7f0bc04c1640 1 -- 172.21.15.155:0/2312013935 shutdown_connections 2024-03-29T11:35:56.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.014+0000 7f0bc04c1640 1 -- 172.21.15.155:0/2312013935 >> 172.21.15.155:0/2312013935 conn(0x7f0bb80b35e0 msgr2=0x7f0bb80b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:56.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.014+0000 7f0bc04c1640 1 -- 172.21.15.155:0/2312013935 shutdown_connections 2024-03-29T11:35:56.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.015+0000 7f0bc04c1640 1 -- 172.21.15.155:0/2312013935 wait complete. 2024-03-29T11:35:56.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:35:56.015+0000 7f0bc04c1640 1 librados: shutdown 2024-03-29T11:35:56.031 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:02.837 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:02.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.980+0000 7fbf19c50640 10 monclient: get_monmap_and_config 2024-03-29T11:36:02.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.980+0000 7fbf19c50640 10 monclient: build_initial_monmap 2024-03-29T11:36:02.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.980+0000 7fbf19c50640 10 monclient: monmap: 2024-03-29T11:36:02.980 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:02.980 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:02.980 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:36:02.982241+0000 2024-03-29T11:36:02.980 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:36:02.982241+0000 2024-03-29T11:36:02.981 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:02.981 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:02.981 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:02.981 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:36:02.981 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:36:02.981 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:02.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.982+0000 7fbf19c50640 1 Processor -- start 2024-03-29T11:36:02.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.982+0000 7fbf19c50640 1 -- start start 2024-03-29T11:36:02.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.982+0000 7fbf19c50640 10 monclient: init 2024-03-29T11:36:02.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.983+0000 7fbf19c50640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:02.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.983+0000 7fbf19c50640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:02.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.983+0000 7fbf19c50640 10 monclient(hunting): picked mon.noname-a con 0x7fbf14071de0 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:02.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.983+0000 7fbf19c50640 10 monclient(hunting): picked mon.noname-b con 0x7fbf140721c0 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:02.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.983+0000 7fbf19c50640 10 monclient(hunting): picked mon.noname-c con 0x7fbf14145ae0 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:02.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.983+0000 7fbf19c50640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf14145650 con 0x7fbf14071de0 2024-03-29T11:36:02.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.983+0000 7fbf19c50640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf140ae260 con 0x7fbf14145ae0 2024-03-29T11:36:02.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.983+0000 7fbf19c50640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf141420d0 con 0x7fbf140721c0 2024-03-29T11:36:02.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.983+0000 7fbf19c50640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:02.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.983+0000 7fbf19c50640 10 monclient(hunting): authenticate will time out at 945.110162s 2024-03-29T11:36:02.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.984+0000 7fbf13fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fbf14145ae0 0x7fbf14145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:57312/0 (socket says 172.21.15.155:57312) 2024-03-29T11:36:02.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.985+0000 7fbf13fff640 1 -- 172.21.15.155:0/423908833 learned_addr learned my addr 172.21.15.155:0/423908833 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:02.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 1 -- 172.21.15.155:0/423908833 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605500438 0 0) 0x7fbf141420d0 con 0x7fbf140721c0 2024-03-29T11:36:02.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:02.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:02.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:02.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:36:02.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 1 -- 172.21.15.155:0/423908833 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbefc003520 con 0x7fbf140721c0 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 1 -- 172.21.15.155:0/423908833 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916393791 0 0) 0x7fbf140ae260 con 0x7fbf14145ae0 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.988+0000 7fbf127fc640 1 -- 172.21.15.155:0/423908833 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf141420d0 con 0x7fbf14145ae0 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 1 -- 172.21.15.155:0/423908833 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278407418 0 0) 0x7fbefc003520 con 0x7fbf140721c0 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 1 -- 172.21.15.155:0/423908833 >> v1:172.21.15.183:6789/0 conn(0x7fbf14145ae0 legacy=0x7fbf14145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 1 -- 172.21.15.155:0/423908833 >> v1:172.21.15.155:6789/0 conn(0x7fbf14071de0 legacy=0x7fbf140aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 1 monclient: found mon.noname-b 2024-03-29T11:36:02.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 1 -- 172.21.15.155:0/423908833 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf140b0590 con 0x7fbf140721c0 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf19c50640 5 monclient: authenticate success, global_id 4280 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf19c50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 1 -- 172.21.15.155:0/423908833 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fbf0c001a10 con 0x7fbf140721c0 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.989+0000 7fbf127fc640 10 monclient: dump: 2024-03-29T11:36:02.992 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.990+0000 7fbf19c50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.990+0000 7fbf127fc640 1 -- 172.21.15.155:0/423908833 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf0c001c30 con 0x7fbf140721c0 2024-03-29T11:36:02.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.990+0000 7fbf127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.990+0000 7fbf19c50640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.990+0000 7fbf19c50640 4 set_mon_vals no callback set 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.991+0000 7fbf127fc640 1 -- 172.21.15.155:0/423908833 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fbf0c001f40 con 0x7fbf140721c0 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.991+0000 7fbf127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.991+0000 7fbf127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.991+0000 7fbf127fc640 10 monclient: dump: 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:02.994 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:02.995 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:02.995 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:02.995 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:02.995 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.991+0000 7fbf19c50640 10 monclient: shutdown 2024-03-29T11:36:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.991+0000 7fbf19c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.991+0000 7fbf19c50640 1 -- 172.21.15.155:0/423908833 >> v1:172.21.15.155:6790/0 conn(0x7fbf140721c0 legacy=0x7fbf1413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.991+0000 7fbf19c50640 1 -- 172.21.15.155:0/423908833 shutdown_connections 2024-03-29T11:36:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.991+0000 7fbf19c50640 1 -- 172.21.15.155:0/423908833 >> 172.21.15.155:0/423908833 conn(0x7fbf1406b660 msgr2=0x7fbf1406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.991+0000 7fbf19c50640 1 -- 172.21.15.155:0/423908833 shutdown_connections 2024-03-29T11:36:02.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.991+0000 7fbf19c50640 1 -- 172.21.15.155:0/423908833 wait complete. 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.992+0000 7fbf19c50640 10 monclient: build_initial_monmap 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.992+0000 7fbf19c50640 10 monclient: monmap: 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:36:02.997 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.992+0000 7fbf19c50640 1 librados: starting msgr at 2024-03-29T11:36:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.992+0000 7fbf19c50640 1 librados: starting objecter 2024-03-29T11:36:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.993+0000 7fbf19c50640 1 Processor -- start 2024-03-29T11:36:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.993+0000 7fbf19c50640 1 -- start start 2024-03-29T11:36:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.993+0000 7fbf19c50640 1 librados: setting wanted keys 2024-03-29T11:36:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.993+0000 7fbf19c50640 1 librados: calling monclient init 2024-03-29T11:36:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.993+0000 7fbf19c50640 10 monclient: init 2024-03-29T11:36:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.994+0000 7fbf19c50640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.994+0000 7fbf19c50640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.994+0000 7fbf19c50640 10 monclient(hunting): picked mon.noname-b con 0x7fbf14071de0 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.994+0000 7fbf19c50640 10 monclient(hunting): picked mon.noname-c con 0x7fbf14145ae0 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.994+0000 7fbf19c50640 10 monclient(hunting): picked mon.noname-a con 0x7fbf140718f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.994+0000 7fbf19c50640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf14136160 con 0x7fbf140718f0 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.994+0000 7fbf19c50640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf14070700 con 0x7fbf14071de0 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.994+0000 7fbf19c50640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf140709e0 con 0x7fbf14145ae0 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.994+0000 7fbf19c50640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.994+0000 7fbf19c50640 10 monclient(hunting): authenticate will time out at 945.121155s 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.996+0000 7fbf137fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fbf14071de0 0x7fbf1406e380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:57920/0 (socket says 172.21.15.155:57920) 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.996+0000 7fbf137fe640 1 -- 172.21.15.155:0/3306776145 learned_addr learned my addr 172.21.15.155:0/3306776145 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.997+0000 7fbefbfff640 10 client.?.objecter ms_handle_connect 0x7fbf14071de0 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.997+0000 7fbefbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.997+0000 7fbefbfff640 10 client.?.objecter ms_handle_connect 0x7fbf14145ae0 2024-03-29T11:36:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.997+0000 7fbefbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.997+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333716039 0 0) 0x7fbf140709e0 con 0x7fbf14145ae0 2024-03-29T11:36:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.997+0000 7fbefbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.997+0000 7fbefbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.997+0000 7fbefbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.997+0000 7fbefbfff640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:36:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.997+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf00004eb0 con 0x7fbf14145ae0 2024-03-29T11:36:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.998+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1330535692 0 0) 0x7fbf00004eb0 con 0x7fbf14145ae0 2024-03-29T11:36:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.998+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf140709e0 con 0x7fbf14145ae0 2024-03-29T11:36:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.998+0000 7fbefbfff640 10 client.?.objecter ms_handle_connect 0x7fbf140718f0 2024-03-29T11:36:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.998+0000 7fbefbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.998+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fbf0c001990 con 0x7fbf14145ae0 2024-03-29T11:36:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.998+0000 7fbefbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.998+0000 7fbefbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:03.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.998+0000 7fbefbfff640 10 monclient(hunting): dump: 2024-03-29T11:36:03.004 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:03.004 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:03.004 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:03.004 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:03.004 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:03.004 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:03.004 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:03.004 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:03.004 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:03.004 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4248482736 0 0) 0x7fbf140709e0 con 0x7fbf14145ae0 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 >> v1:172.21.15.183:6789/0 conn(0x7fbf14071de0 legacy=0x7fbf1406e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 >> v1:172.21.15.155:6789/0 conn(0x7fbf140718f0 legacy=0x7fbf141fff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 1 monclient: found mon.c 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1406f8a0 con 0x7fbf14145ae0 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 10 monclient: _finish_auth 0 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:03.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:03.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbf19c50640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:36:03.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbf19c50640 10 monclient: _renew_subs 2024-03-29T11:36:03.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbf19c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:03.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbf19c50640 1 -- 172.21.15.155:0/3306776145 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf1406ea90 con 0x7fbf14145ae0 2024-03-29T11:36:03.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbf19c50640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:03.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbf19c50640 10 monclient: _renew_subs 2024-03-29T11:36:03.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbf19c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:03.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbf19c50640 1 -- 172.21.15.155:0/3306776145 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1406f4c0 con 0x7fbf14145ae0 2024-03-29T11:36:03.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbf19c50640 1 librados: init done 2024-03-29T11:36:03.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf0c003c20 con 0x7fbf14145ae0 2024-03-29T11:36:03.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fbf0c003fb0 con 0x7fbf14145ae0 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbefbfff640 10 monclient: dump: 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:03.007 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:02.999+0000 7fbf11ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.001+0000 7fbf19c50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.001+0000 7fbf19c50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.001+0000 7fbf19c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.001+0000 7fbf19c50640 1 -- 172.21.15.155:0/3306776145 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef4005080 con 0x7fbf14145ae0 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.002+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fbf0c004110 con 0x7fbf14145ae0 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.002+0000 7fbf137fe640 10 monclient: get_auth_request con 0x7fbf0006dd60 auth_method 0 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.002+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fbf0c083cd0 con 0x7fbf14145ae0 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.002+0000 7fbefbfff640 10 client.4283.objecter ms_dispatch 0x7fbf14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.002+0000 7fbefbfff640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.002+0000 7fbefbfff640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.002+0000 7fbefbfff640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-29T11:36:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.002+0000 7fbefbfff640 10 client.4283.objecter ms_handle_connect 0x7fbf0006dd60 2024-03-29T11:36:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.010+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf0c04f5b0 con 0x7fbf14145ae0 2024-03-29T11:36:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.010+0000 7fbefbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.010+0000 7fbefbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:03.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.331+0000 7fbf19c50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:03.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.331+0000 7fbf19c50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:03.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.331+0000 7fbf19c50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:03.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.331+0000 7fbf19c50640 1 -- 172.21.15.155:0/3306776145 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbef4005330 con 0x7fbf14145ae0 2024-03-29T11:36:03.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.332+0000 7fbefbfff640 1 -- 172.21.15.155:0/3306776145 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fbf0c0514e0 con 0x7fbf14145ae0 2024-03-29T11:36:03.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.332+0000 7fbefbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:03.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.332+0000 7fbefbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:03.333 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:36:03.333 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:03.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.338+0000 7fbf19c50640 10 librados: watch_flush enter 2024-03-29T11:36:03.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.338+0000 7fbf19c50640 10 librados: watch_flush exit 2024-03-29T11:36:03.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.338+0000 7fbf19c50640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-29T11:36:03.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.338+0000 7fbf19c50640 10 client.4283.objecter successfully canceled tick 2024-03-29T11:36:03.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.338+0000 7fbf19c50640 1 -- 172.21.15.155:0/3306776145 >> v1:172.21.15.155:6816/1760814708 conn(0x7fbf0006dd60 legacy=0x7fbf00070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.338+0000 7fbf19c50640 10 monclient: shutdown 2024-03-29T11:36:03.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.338+0000 7fbf19c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:03.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.338+0000 7fbf19c50640 1 -- 172.21.15.155:0/3306776145 >> v1:172.21.15.155:6790/0 conn(0x7fbf14145ae0 legacy=0x7fbf140711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.339+0000 7fbf19c50640 1 -- 172.21.15.155:0/3306776145 shutdown_connections 2024-03-29T11:36:03.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.339+0000 7fbf19c50640 1 -- 172.21.15.155:0/3306776145 >> 172.21.15.155:0/3306776145 conn(0x7fbf1406b660 msgr2=0x7fbf1406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:03.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.340+0000 7fbf19c50640 1 -- 172.21.15.155:0/3306776145 shutdown_connections 2024-03-29T11:36:03.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.340+0000 7fbf19c50640 1 -- 172.21.15.155:0/3306776145 wait complete. 2024-03-29T11:36:03.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:03.340+0000 7fbf19c50640 1 librados: shutdown 2024-03-29T11:36:03.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:10.164 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.313+0000 7fd11bea8640 10 monclient: get_monmap_and_config 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.313+0000 7fd11bea8640 10 monclient: build_initial_monmap 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.313+0000 7fd11bea8640 10 monclient: monmap: 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:36:10.315262+0000 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:36:10.315262+0000 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:36:10.313 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:36:10.314 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:10.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.314+0000 7fd11bea8640 1 Processor -- start 2024-03-29T11:36:10.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 1 -- start start 2024-03-29T11:36:10.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 10 monclient: init 2024-03-29T11:36:10.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:10.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:10.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 10 monclient(hunting): picked mon.noname-c con 0x7fd1140a9750 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:10.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 10 monclient(hunting): picked mon.noname-a con 0x7fd11413e1e0 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:10.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 10 monclient(hunting): picked mon.noname-b con 0x7fd114071c30 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:10.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1140ab300 con 0x7fd11413e1e0 2024-03-29T11:36:10.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1140ac2f0 con 0x7fd1140a9750 2024-03-29T11:36:10.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1140ad8d0 con 0x7fd114071c30 2024-03-29T11:36:10.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:10.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.315+0000 7fd11bea8640 10 monclient(hunting): authenticate will time out at 952.442800s 2024-03-29T11:36:10.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.319+0000 7fd119443640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd11413e1e0 0x7fd1140ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:56503/0 (socket says 172.21.15.155:56503) 2024-03-29T11:36:10.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.319+0000 7fd119443640 1 -- 172.21.15.155:0/1603794838 learned_addr learned my addr 172.21.15.155:0/1603794838 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:10.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.321+0000 7fd118c42640 1 -- 172.21.15.155:0/1603794838 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622747366 0 0) 0x7fd1140ab300 con 0x7fd11413e1e0 2024-03-29T11:36:10.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.321+0000 7fd118c42640 10 monclient(hunting): _negotiate 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.321+0000 7fd118c42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.321+0000 7fd118c42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.321+0000 7fd118c42640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.321+0000 7fd118c42640 1 -- 172.21.15.155:0/1603794838 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0fc003520 con 0x7fd11413e1e0 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.322+0000 7fd118c42640 1 -- 172.21.15.155:0/1603794838 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2132250863 0 0) 0x7fd0fc003520 con 0x7fd11413e1e0 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.322+0000 7fd118c42640 1 -- 172.21.15.155:0/1603794838 >> v1:172.21.15.155:6790/0 conn(0x7fd114071c30 legacy=0x7fd1140ace20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.322+0000 7fd118c42640 1 -- 172.21.15.155:0/1603794838 >> v1:172.21.15.183:6789/0 conn(0x7fd1140a9750 legacy=0x7fd11413dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.322+0000 7fd118c42640 10 monclient: _finish_hunting 0 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.322+0000 7fd118c42640 1 monclient: found mon.noname-a 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.322+0000 7fd118c42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.322+0000 7fd118c42640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.322+0000 7fd118c42640 1 -- 172.21.15.155:0/1603794838 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1140ada60 con 0x7fd11413e1e0 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.322+0000 7fd118c42640 10 monclient: _finish_auth 0 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.322+0000 7fd118c42640 10 monclient: _check_auth_tickets 2024-03-29T11:36:10.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.322+0000 7fd118c42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.323+0000 7fd11bea8640 5 monclient: authenticate success, global_id 4284 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.323+0000 7fd11bea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.323+0000 7fd118c42640 1 -- 172.21.15.155:0/1603794838 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd108001690 con 0x7fd11413e1e0 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.323+0000 7fd118c42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.323+0000 7fd118c42640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.323+0000 7fd118c42640 10 monclient: dump: 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:10.325 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:10.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.323+0000 7fd11bea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:10.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.323+0000 7fd118c42640 1 -- 172.21.15.155:0/1603794838 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd108001920 con 0x7fd11413e1e0 2024-03-29T11:36:10.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.323+0000 7fd118c42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:10.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.323+0000 7fd11bea8640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:10.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.323+0000 7fd11bea8640 4 set_mon_vals no callback set 2024-03-29T11:36:10.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.324+0000 7fd11bea8640 10 monclient: shutdown 2024-03-29T11:36:10.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.324+0000 7fd11bea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:10.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.324+0000 7fd11bea8640 1 -- 172.21.15.155:0/1603794838 >> v1:172.21.15.155:6789/0 conn(0x7fd11413e1e0 legacy=0x7fd1140ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.324+0000 7fd11bea8640 1 -- 172.21.15.155:0/1603794838 shutdown_connections 2024-03-29T11:36:10.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.324+0000 7fd11bea8640 1 -- 172.21.15.155:0/1603794838 >> 172.21.15.155:0/1603794838 conn(0x7fd11406b660 msgr2=0x7fd11406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:10.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.327+0000 7fd11bea8640 1 -- 172.21.15.155:0/1603794838 shutdown_connections 2024-03-29T11:36:10.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.327+0000 7fd11bea8640 1 -- 172.21.15.155:0/1603794838 wait complete. 2024-03-29T11:36:10.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.328+0000 7fd11bea8640 10 monclient: build_initial_monmap 2024-03-29T11:36:10.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.328+0000 7fd11bea8640 10 monclient: monmap: 2024-03-29T11:36:10.328 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:10.328 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:10.328 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.328+0000 7fd11bea8640 1 librados: starting msgr at 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.328+0000 7fd11bea8640 1 librados: starting objecter 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.328+0000 7fd11bea8640 1 Processor -- start 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.329+0000 7fd11bea8640 1 -- start start 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.329+0000 7fd11bea8640 1 librados: setting wanted keys 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.329+0000 7fd11bea8640 1 librados: calling monclient init 2024-03-29T11:36:10.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.329+0000 7fd11bea8640 10 monclient: init 2024-03-29T11:36:10.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.330+0000 7fd11bea8640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:10.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.330+0000 7fd11bea8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:10.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.330+0000 7fd11bea8640 10 monclient(hunting): picked mon.noname-a con 0x7fd114071c30 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:10.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.330+0000 7fd11bea8640 10 monclient(hunting): picked mon.noname-b con 0x7fd1140a9750 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:10.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.330+0000 7fd11bea8640 10 monclient(hunting): picked mon.noname-c con 0x7fd11413e1e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:10.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.330+0000 7fd11bea8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11413b0d0 con 0x7fd114071c30 2024-03-29T11:36:10.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.330+0000 7fd11bea8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11406ea50 con 0x7fd1140a9750 2024-03-29T11:36:10.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.330+0000 7fd11bea8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11406ed30 con 0x7fd11413e1e0 2024-03-29T11:36:10.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.330+0000 7fd11bea8640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:10.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.330+0000 7fd11bea8640 10 monclient(hunting): authenticate will time out at 952.457383s 2024-03-29T11:36:10.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.334+0000 7fd119c44640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd114071c30 0x7fd1141e4790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:56621/0 (socket says 172.21.15.155:56621) 2024-03-29T11:36:10.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.334+0000 7fd119c44640 1 -- 172.21.15.155:0/1697608324 learned_addr learned my addr 172.21.15.155:0/1697608324 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:10.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.334+0000 7fd1127fc640 10 client.?.objecter ms_handle_connect 0x7fd114071c30 2024-03-29T11:36:10.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.334+0000 7fd1127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:10.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 10 client.?.objecter ms_handle_connect 0x7fd1140a9750 2024-03-29T11:36:10.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:10.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548971485 0 0) 0x7fd11413b0d0 con 0x7fd114071c30 2024-03-29T11:36:10.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:10.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:10.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:10.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:36:10.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd100004eb0 con 0x7fd114071c30 2024-03-29T11:36:10.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570251182 0 0) 0x7fd11406ea50 con 0x7fd1140a9750 2024-03-29T11:36:10.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:10.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:10.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:10.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:36:10.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.337+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd11413b0d0 con 0x7fd1140a9750 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439879293 0 0) 0x7fd11413b0d0 con 0x7fd1140a9750 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd11406ea50 con 0x7fd1140a9750 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1882418261 0 0) 0x7fd100004eb0 con 0x7fd114071c30 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd11413b0d0 con 0x7fd114071c30 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 client.?.objecter ms_handle_connect 0x7fd11413e1e0 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd108001020 con 0x7fd1140a9750 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 monclient(hunting): dump: 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:10.338 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd10c001950 con 0x7fd114071c30 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 monclient(hunting): dump: 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:10.339 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 573269417 0 0) 0x7fd11406ea50 con 0x7fd1140a9750 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 >> v1:172.21.15.155:6790/0 conn(0x7fd11413e1e0 legacy=0x7fd11413a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 >> v1:172.21.15.155:6789/0 conn(0x7fd114071c30 legacy=0x7fd1141e4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 1 monclient: found mon.b 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11406f010 con 0x7fd1140a9750 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.338+0000 7fd1127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.339+0000 7fd11bea8640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.339+0000 7fd11bea8640 10 monclient: _renew_subs 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.339+0000 7fd11bea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.339+0000 7fd11bea8640 1 -- 172.21.15.155:0/1697608324 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1140b1190 con 0x7fd1140a9750 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.339+0000 7fd11bea8640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.339+0000 7fd11bea8640 10 monclient: _renew_subs 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.339+0000 7fd11bea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.339+0000 7fd11bea8640 1 -- 172.21.15.155:0/1697608324 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1141e5080 con 0x7fd1140a9750 2024-03-29T11:36:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.339+0000 7fd11bea8640 1 librados: init done 2024-03-29T11:36:10.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.341+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd108001b10 con 0x7fd1140a9750 2024-03-29T11:36:10.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.341+0000 7fd1127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:10.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.341+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd108004fb0 con 0x7fd1140a9750 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.341+0000 7fd1127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.341+0000 7fd1127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.341+0000 7fd1127fc640 10 monclient: dump: 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.342+0000 7fd113fff640 4 set_mon_vals no callback set 2024-03-29T11:36:10.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.342+0000 7fd11bea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:10.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.343+0000 7fd11bea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:10.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.343+0000 7fd11bea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:36:10.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.343+0000 7fd11bea8640 1 -- 172.21.15.155:0/1697608324 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f4005080 con 0x7fd1140a9750 2024-03-29T11:36:10.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.345+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fd10801af80 con 0x7fd1140a9750 2024-03-29T11:36:10.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.346+0000 7fd119c44640 10 monclient: get_auth_request con 0x7fd10006dea0 auth_method 0 2024-03-29T11:36:10.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.346+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fd108084bc0 con 0x7fd1140a9750 2024-03-29T11:36:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.346+0000 7fd1127fc640 10 client.4291.objecter ms_dispatch 0x7fd114000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.346+0000 7fd1127fc640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.346+0000 7fd1127fc640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.346+0000 7fd1127fc640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T11:36:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.346+0000 7fd1127fc640 10 client.4291.objecter ms_handle_connect 0x7fd10006dea0 2024-03-29T11:36:10.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.352+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1080504a0 con 0x7fd1140a9750 2024-03-29T11:36:10.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.352+0000 7fd1127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:10.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.352+0000 7fd1127fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:10.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.674+0000 7fd11bea8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:10.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.674+0000 7fd11bea8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:10.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.674+0000 7fd11bea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:36:10.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.674+0000 7fd11bea8640 1 -- 172.21.15.155:0/1697608324 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0f4005330 con 0x7fd1140a9750 2024-03-29T11:36:10.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.677+0000 7fd1127fc640 1 -- 172.21.15.155:0/1697608324 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fd1080523d0 con 0x7fd1140a9750 2024-03-29T11:36:10.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.677+0000 7fd1127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:10.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.677+0000 7fd1127fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:10.676 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:36:10.676 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:10.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.680+0000 7fd11bea8640 10 librados: watch_flush enter 2024-03-29T11:36:10.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.680+0000 7fd11bea8640 10 librados: watch_flush exit 2024-03-29T11:36:10.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.680+0000 7fd11bea8640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T11:36:10.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.680+0000 7fd11bea8640 10 client.4291.objecter successfully canceled tick 2024-03-29T11:36:10.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.680+0000 7fd11bea8640 1 -- 172.21.15.155:0/1697608324 >> v1:172.21.15.155:6816/1760814708 conn(0x7fd10006dea0 legacy=0x7fd100070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.680+0000 7fd11bea8640 10 monclient: shutdown 2024-03-29T11:36:10.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.680+0000 7fd11bea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:10.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.680+0000 7fd11bea8640 1 -- 172.21.15.155:0/1697608324 >> v1:172.21.15.183:6789/0 conn(0x7fd1140a9750 legacy=0x7fd11413a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.681+0000 7fd11bea8640 1 -- 172.21.15.155:0/1697608324 shutdown_connections 2024-03-29T11:36:10.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.681+0000 7fd11bea8640 1 -- 172.21.15.155:0/1697608324 >> 172.21.15.155:0/1697608324 conn(0x7fd11406b660 msgr2=0x7fd1140b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:10.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.681+0000 7fd11bea8640 1 -- 172.21.15.155:0/1697608324 shutdown_connections 2024-03-29T11:36:10.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.681+0000 7fd11bea8640 1 -- 172.21.15.155:0/1697608324 wait complete. 2024-03-29T11:36:10.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:10.681+0000 7fd11bea8640 1 librados: shutdown 2024-03-29T11:36:10.700 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:17.506 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:17.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.659+0000 7f4bf357b640 10 monclient: get_monmap_and_config 2024-03-29T11:36:17.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.659+0000 7f4bf357b640 10 monclient: build_initial_monmap 2024-03-29T11:36:17.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.659+0000 7f4bf357b640 10 monclient: monmap: 2024-03-29T11:36:17.659 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:17.659 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:17.659 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:36:17.660009+0000 2024-03-29T11:36:17.659 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:36:17.660009+0000 2024-03-29T11:36:17.659 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:17.659 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:17.659 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:17.660 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:36:17.660 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:36:17.660 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:17.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.660+0000 7f4bf357b640 1 Processor -- start 2024-03-29T11:36:17.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.661+0000 7f4bf357b640 1 -- start start 2024-03-29T11:36:17.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.661+0000 7f4bf357b640 10 monclient: init 2024-03-29T11:36:17.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.661+0000 7f4bf357b640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:17.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.661+0000 7f4bf357b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:17.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.661+0000 7f4bf357b640 10 monclient(hunting): picked mon.noname-a con 0x7f4bec0acf60 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:17.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.662+0000 7f4bf357b640 10 monclient(hunting): picked mon.noname-c con 0x7f4bec13ff40 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:17.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.662+0000 7f4bf357b640 10 monclient(hunting): picked mon.noname-b con 0x7f4bec1403f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:17.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.662+0000 7f4bf357b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec0b4310 con 0x7f4bec0acf60 2024-03-29T11:36:17.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.662+0000 7f4bf357b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec140800 con 0x7f4bec13ff40 2024-03-29T11:36:17.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.662+0000 7f4bf357b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec0aec00 con 0x7f4bec1403f0 2024-03-29T11:36:17.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.662+0000 7f4bf357b640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:17.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.662+0000 7f4bf357b640 10 monclient(hunting): authenticate will time out at 959.787933s 2024-03-29T11:36:17.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.665+0000 7f4bf0b16640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f4bec13ff40 0x7f4bec148840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:37725/0 (socket says 172.21.15.155:37725) 2024-03-29T11:36:17.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.665+0000 7f4bf0b16640 1 -- 172.21.15.155:0/3383913879 learned_addr learned my addr 172.21.15.155:0/3383913879 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:17.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.667+0000 7f4be3fff640 1 -- 172.21.15.155:0/3383913879 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414717374 0 0) 0x7f4bec140800 con 0x7f4bec13ff40 2024-03-29T11:36:17.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.667+0000 7f4be3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:17.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.667+0000 7f4be3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:17.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.667+0000 7f4be3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:17.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.667+0000 7f4be3fff640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:36:17.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.667+0000 7f4be3fff640 1 -- 172.21.15.155:0/3383913879 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd8003520 con 0x7f4bec13ff40 2024-03-29T11:36:17.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.668+0000 7f4be3fff640 1 -- 172.21.15.155:0/3383913879 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2566401211 0 0) 0x7f4bd8003520 con 0x7f4bec13ff40 2024-03-29T11:36:17.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.668+0000 7f4be3fff640 1 -- 172.21.15.155:0/3383913879 >> v1:172.21.15.155:6790/0 conn(0x7f4bec1403f0 legacy=0x7f4bec14bf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:17.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.668+0000 7f4be3fff640 1 -- 172.21.15.155:0/3383913879 >> v1:172.21.15.155:6789/0 conn(0x7f4bec0acf60 legacy=0x7f4bec139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.668+0000 7f4be3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.668+0000 7f4be3fff640 1 monclient: found mon.noname-c 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 1 -- 172.21.15.155:0/3383913879 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec0aeee0 con 0x7f4bec13ff40 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 1 -- 172.21.15.155:0/3383913879 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4bdc001690 con 0x7f4bec13ff40 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 10 monclient: dump: 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:17.669 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:17.670 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:17.670 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:17.670 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:17.670 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:17.670 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:17.670 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:17.670 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:17.670 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:17.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4bf357b640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:36:17.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4bf357b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 1 -- 172.21.15.155:0/3383913879 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bdc001860 con 0x7f4bec13ff40 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 1 -- 172.21.15.155:0/3383913879 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4bdc001cb0 con 0x7f4bec13ff40 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4be3fff640 10 monclient: dump: 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:17.671 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:17.672 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:17.672 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:17.672 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:17.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4bf357b640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:17.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.669+0000 7f4bf357b640 4 set_mon_vals no callback set 2024-03-29T11:36:17.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.670+0000 7f4bf357b640 10 monclient: shutdown 2024-03-29T11:36:17.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.670+0000 7f4bf357b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:17.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.670+0000 7f4bf357b640 1 -- 172.21.15.155:0/3383913879 >> v1:172.21.15.183:6789/0 conn(0x7f4bec13ff40 legacy=0x7f4bec148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.671+0000 7f4bf357b640 1 -- 172.21.15.155:0/3383913879 shutdown_connections 2024-03-29T11:36:17.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.671+0000 7f4bf357b640 1 -- 172.21.15.155:0/3383913879 >> 172.21.15.155:0/3383913879 conn(0x7f4bec0b35e0 msgr2=0x7f4bec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:17.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.673+0000 7f4bf357b640 1 -- 172.21.15.155:0/3383913879 shutdown_connections 2024-03-29T11:36:17.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.673+0000 7f4bf357b640 1 -- 172.21.15.155:0/3383913879 wait complete. 2024-03-29T11:36:17.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.674+0000 7f4bf357b640 10 monclient: build_initial_monmap 2024-03-29T11:36:17.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.674+0000 7f4bf357b640 10 monclient: monmap: 2024-03-29T11:36:17.675 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.675+0000 7f4bf357b640 1 librados: starting msgr at 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.675+0000 7f4bf357b640 1 librados: starting objecter 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.675+0000 7f4bf357b640 1 Processor -- start 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.675+0000 7f4bf357b640 1 -- start start 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.675+0000 7f4bf357b640 1 librados: setting wanted keys 2024-03-29T11:36:17.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.675+0000 7f4bf357b640 1 librados: calling monclient init 2024-03-29T11:36:17.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.675+0000 7f4bf357b640 10 monclient: init 2024-03-29T11:36:17.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.676+0000 7f4bf357b640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:17.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.676+0000 7f4bf357b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:17.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.678+0000 7f4bf357b640 10 monclient(hunting): picked mon.noname-c con 0x7f4bec0acf60 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:17.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.678+0000 7f4bf357b640 10 monclient(hunting): picked mon.noname-b con 0x7f4bec13ff40 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:17.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.678+0000 7f4bf357b640 10 monclient(hunting): picked mon.noname-a con 0x7f4bec1403f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:17.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.678+0000 7f4bf357b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec1d2fe0 con 0x7f4bec1403f0 2024-03-29T11:36:17.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.678+0000 7f4bf357b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec1c9340 con 0x7f4bec13ff40 2024-03-29T11:36:17.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.678+0000 7f4bf357b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec1d3b80 con 0x7f4bec0acf60 2024-03-29T11:36:17.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.678+0000 7f4bf357b640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:17.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.678+0000 7f4bf357b640 10 monclient(hunting): authenticate will time out at 959.804080s 2024-03-29T11:36:17.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.683+0000 7f4bf0b16640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f4bec13ff40 0x7f4bec133790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:37833/0 (socket says 172.21.15.155:37833) 2024-03-29T11:36:17.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.683+0000 7f4bf0b16640 1 -- 172.21.15.155:0/485668055 learned_addr learned my addr 172.21.15.155:0/485668055 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:17.682 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.684+0000 7f4be1ffb640 10 client.?.objecter ms_handle_connect 0x7f4bec13ff40 2024-03-29T11:36:17.682 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.684+0000 7f4be1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:17.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.685+0000 7f4be1ffb640 10 client.?.objecter ms_handle_connect 0x7f4bec1403f0 2024-03-29T11:36:17.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.685+0000 7f4be1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:17.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.687+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760014268 0 0) 0x7f4bec1d2fe0 con 0x7f4bec1403f0 2024-03-29T11:36:17.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.687+0000 7f4be1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:17.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.687+0000 7f4be1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:17.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.687+0000 7f4be1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:17.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.687+0000 7f4be1ffb640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:36:17.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.687+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd0004eb0 con 0x7f4bec1403f0 2024-03-29T11:36:17.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.687+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782689794 0 0) 0x7f4bec1c9340 con 0x7f4bec13ff40 2024-03-29T11:36:17.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:17.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:17.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:17.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bec1d2fe0 con 0x7f4bec13ff40 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1588782246 0 0) 0x7f4bd0004eb0 con 0x7f4bec1403f0 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bec1c9340 con 0x7f4bec1403f0 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4be4001750 con 0x7f4bec1403f0 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:17.694 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.688+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3305929394 0 0) 0x7f4bec1d2fe0 con 0x7f4bec13ff40 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bd0004eb0 con 0x7f4bec13ff40 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4017512862 0 0) 0x7f4bec1c9340 con 0x7f4bec1403f0 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 >> v1:172.21.15.155:6790/0 conn(0x7f4bec0acf60 legacy=0x7f4bec1d3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 >> v1:172.21.15.183:6789/0 conn(0x7f4bec13ff40 legacy=0x7f4bec133790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 1 monclient: found mon.a 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec133ea0 con 0x7f4bec1403f0 2024-03-29T11:36:17.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.689+0000 7f4be1ffb640 10 client.?.objecter ms_handle_connect 0x7f4bec0acf60 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.690+0000 7f4bf357b640 5 monclient: authenticate success, global_id 4290 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.690+0000 7f4bf357b640 10 monclient: _renew_subs 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.690+0000 7f4bf357b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.690+0000 7f4bf357b640 1 -- 172.21.15.155:0/485668055 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bec1340f0 con 0x7f4bec1403f0 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.690+0000 7f4bf357b640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.690+0000 7f4bf357b640 10 monclient: _renew_subs 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.690+0000 7f4bf357b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.690+0000 7f4bf357b640 1 -- 172.21.15.155:0/485668055 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bec0ab270 con 0x7f4bec1403f0 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.690+0000 7f4bf357b640 1 librados: init done 2024-03-29T11:36:17.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.691+0000 7f4be1ffb640 10 client.4290.objecter resend_mon_ops 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.691+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4be40018b0 con 0x7f4bec1403f0 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.691+0000 7f4be1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.691+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4be4003790 con 0x7f4bec1403f0 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.692+0000 7f4be1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.692+0000 7f4be1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.692+0000 7f4be1ffb640 10 monclient: dump: 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:17.697 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:17.698 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:17.698 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:17.698 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:17.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.692+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f4be4019760 con 0x7f4bec1403f0 2024-03-29T11:36:17.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.692+0000 7f4be37fe640 4 set_mon_vals no callback set 2024-03-29T11:36:17.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.693+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f4be40836c0 con 0x7f4bec1403f0 2024-03-29T11:36:17.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.693+0000 7f4be1ffb640 10 client.4290.objecter ms_dispatch 0x7f4bec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:17.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.693+0000 7f4be1ffb640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:17.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.693+0000 7f4be1ffb640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:17.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.693+0000 7f4be1ffb640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-29T11:36:17.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.693+0000 7f4bf1317640 10 monclient: get_auth_request con 0x7f4bd006de30 auth_method 0 2024-03-29T11:36:17.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.694+0000 7f4be1ffb640 10 client.4290.objecter ms_handle_connect 0x7f4bd006de30 2024-03-29T11:36:17.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.694+0000 7f4bf357b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:17.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.694+0000 7f4bf357b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:17.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.694+0000 7f4bf357b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:17.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.694+0000 7f4bf357b640 1 -- 172.21.15.155:0/485668055 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bcc005080 con 0x7f4bec1403f0 2024-03-29T11:36:17.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.703+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4be404efa0 con 0x7f4bec1403f0 2024-03-29T11:36:17.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.703+0000 7f4be1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:17.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:17.703+0000 7f4be1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.026+0000 7f4bf357b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.026+0000 7f4bf357b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.026+0000 7f4bf357b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.027+0000 7f4bf357b640 1 -- 172.21.15.155:0/485668055 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bcc005330 con 0x7f4bec1403f0 2024-03-29T11:36:18.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.028+0000 7f4be1ffb640 1 -- 172.21.15.155:0/485668055 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f4be4050ed0 con 0x7f4bec1403f0 2024-03-29T11:36:18.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.028+0000 7f4be1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:18.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.028+0000 7f4be1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:18.030 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:36:18.030 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:18.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.036+0000 7f4bf357b640 10 librados: watch_flush enter 2024-03-29T11:36:18.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.036+0000 7f4bf357b640 10 librados: watch_flush exit 2024-03-29T11:36:18.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.036+0000 7f4bf357b640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-29T11:36:18.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.036+0000 7f4bf357b640 10 client.4290.objecter successfully canceled tick 2024-03-29T11:36:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.037+0000 7f4bf357b640 1 -- 172.21.15.155:0/485668055 >> v1:172.21.15.155:6816/1760814708 conn(0x7f4bd006de30 legacy=0x7f4bd00702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.037+0000 7f4bf357b640 10 monclient: shutdown 2024-03-29T11:36:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.037+0000 7f4bf357b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.037+0000 7f4bf357b640 1 -- 172.21.15.155:0/485668055 >> v1:172.21.15.155:6789/0 conn(0x7f4bec1403f0 legacy=0x7f4bec1c7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.041+0000 7f4bf357b640 1 -- 172.21.15.155:0/485668055 shutdown_connections 2024-03-29T11:36:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.041+0000 7f4bf357b640 1 -- 172.21.15.155:0/485668055 >> 172.21.15.155:0/485668055 conn(0x7f4bec0b35e0 msgr2=0x7f4bec0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.041+0000 7f4bf357b640 1 -- 172.21.15.155:0/485668055 shutdown_connections 2024-03-29T11:36:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.041+0000 7f4bf357b640 1 -- 172.21.15.155:0/485668055 wait complete. 2024-03-29T11:36:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:18.041+0000 7f4bf357b640 1 librados: shutdown 2024-03-29T11:36:18.058 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:24.865 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:25.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.018+0000 7f79a14c1640 10 monclient: get_monmap_and_config 2024-03-29T11:36:25.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.018+0000 7f79a14c1640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.018+0000 7f79a14c1640 10 monclient: monmap: 2024-03-29T11:36:25.016 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:25.017 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:25.017 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:36:25.018505+0000 2024-03-29T11:36:25.017 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:36:25.018505+0000 2024-03-29T11:36:25.017 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.017 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:25.017 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:25.017 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:36:25.017 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:36:25.017 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:25.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.020+0000 7f79a14c1640 1 Processor -- start 2024-03-29T11:36:25.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.020+0000 7f79a14c1640 1 -- start start 2024-03-29T11:36:25.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.020+0000 7f79a14c1640 10 monclient: init 2024-03-29T11:36:25.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.021+0000 7f79a14c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.021+0000 7f79a14c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:25.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.021+0000 7f79a14c1640 10 monclient(hunting): picked mon.noname-a con 0x7f799c1457f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:25.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.025+0000 7f79a14c1640 10 monclient(hunting): picked mon.noname-c con 0x7f799c13f650 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:25.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.025+0000 7f79a14c1640 10 monclient(hunting): picked mon.noname-b con 0x7f799c0a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:25.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.025+0000 7f79a14c1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c0aaef0 con 0x7f799c1457f0 2024-03-29T11:36:25.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.025+0000 7f79a14c1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c145bd0 con 0x7f799c13f650 2024-03-29T11:36:25.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.025+0000 7f79a14c1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c145eb0 con 0x7f799c0a9d70 2024-03-29T11:36:25.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.025+0000 7f79a14c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.025+0000 7f79a14c1640 10 monclient(hunting): authenticate will time out at 967.151568s 2024-03-29T11:36:25.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.025+0000 7f799affd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f799c1457f0 0x7f799c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:41415/0 (socket says 172.21.15.155:41415) 2024-03-29T11:36:25.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.025+0000 7f799affd640 1 -- 172.21.15.155:0/1688772705 learned_addr learned my addr 172.21.15.155:0/1688772705 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:25.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.029+0000 7f799a7fc640 1 -- 172.21.15.155:0/1688772705 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267008079 0 0) 0x7f799c0aaef0 con 0x7f799c1457f0 2024-03-29T11:36:25.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 1 -- 172.21.15.155:0/1688772705 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7988003520 con 0x7f799c1457f0 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 1 -- 172.21.15.155:0/1688772705 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179035118 0 0) 0x7f799c145eb0 con 0x7f799c0a9d70 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 1 -- 172.21.15.155:0/1688772705 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f799c0aaef0 con 0x7f799c0a9d70 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 1 -- 172.21.15.155:0/1688772705 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4244247638 0 0) 0x7f7988003520 con 0x7f799c1457f0 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 1 -- 172.21.15.155:0/1688772705 >> v1:172.21.15.155:6790/0 conn(0x7f799c0a9d70 legacy=0x7f799c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 1 -- 172.21.15.155:0/1688772705 >> v1:172.21.15.183:6789/0 conn(0x7f799c13f650 legacy=0x7f799c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.030+0000 7f799a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 1 -- 172.21.15.155:0/1688772705 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c146340 con 0x7f799c1457f0 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 1 -- 172.21.15.155:0/1688772705 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f798c001690 con 0x7f799c1457f0 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 10 monclient: dump: 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:25.033 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f79a14c1640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f79a14c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 1 -- 172.21.15.155:0/1688772705 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f798c0018b0 con 0x7f799c1457f0 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.031+0000 7f799a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.032+0000 7f799a7fc640 1 -- 172.21.15.155:0/1688772705 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f798c001c40 con 0x7f799c1457f0 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.032+0000 7f799a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.032+0000 7f799a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.032+0000 7f799a7fc640 10 monclient: dump: 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:25.034 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.032+0000 7f79a14c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.032+0000 7f79a14c1640 4 set_mon_vals no callback set 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.033+0000 7f79a14c1640 10 monclient: shutdown 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.033+0000 7f79a14c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.033+0000 7f79a14c1640 1 -- 172.21.15.155:0/1688772705 >> v1:172.21.15.155:6789/0 conn(0x7f799c1457f0 legacy=0x7f799c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.033+0000 7f79a14c1640 1 -- 172.21.15.155:0/1688772705 shutdown_connections 2024-03-29T11:36:25.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.033+0000 7f79a14c1640 1 -- 172.21.15.155:0/1688772705 >> 172.21.15.155:0/1688772705 conn(0x7f799c066a00 msgr2=0x7f799c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.036+0000 7f79a14c1640 1 -- 172.21.15.155:0/1688772705 shutdown_connections 2024-03-29T11:36:25.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.036+0000 7f79a14c1640 1 -- 172.21.15.155:0/1688772705 wait complete. 2024-03-29T11:36:25.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.037+0000 7f79a14c1640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.038+0000 7f79a14c1640 10 monclient: monmap: 2024-03-29T11:36:25.037 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:25.037 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:25.037 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:36:25.037 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:36:25.037 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.037 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:25.037 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:25.037 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:36:25.037 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:36:25.037 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:25.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.038+0000 7f79a14c1640 1 librados: starting msgr at 2024-03-29T11:36:25.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.038+0000 7f79a14c1640 1 librados: starting objecter 2024-03-29T11:36:25.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.039+0000 7f79a14c1640 1 Processor -- start 2024-03-29T11:36:25.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.039+0000 7f79a14c1640 1 -- start start 2024-03-29T11:36:25.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.039+0000 7f79a14c1640 1 librados: setting wanted keys 2024-03-29T11:36:25.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.039+0000 7f79a14c1640 1 librados: calling monclient init 2024-03-29T11:36:25.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.040+0000 7f79a14c1640 10 monclient: init 2024-03-29T11:36:25.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.040+0000 7f79a14c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.041+0000 7f79a14c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:25.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.041+0000 7f79a14c1640 10 monclient(hunting): picked mon.noname-c con 0x7f799c0a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:25.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.042+0000 7f79a14c1640 10 monclient(hunting): picked mon.noname-a con 0x7f799c13f650 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:25.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.042+0000 7f79a14c1640 10 monclient(hunting): picked mon.noname-b con 0x7f799c1457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:25.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.042+0000 7f79a14c1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c0afed0 con 0x7f799c13f650 2024-03-29T11:36:25.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.042+0000 7f79a14c1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c1d70d0 con 0x7f799c1457f0 2024-03-29T11:36:25.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.042+0000 7f79a14c1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c0aed90 con 0x7f799c0a9d70 2024-03-29T11:36:25.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.042+0000 7f79a14c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.042+0000 7f79a14c1640 10 monclient(hunting): authenticate will time out at 967.168396s 2024-03-29T11:36:25.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.047+0000 7f79927fc640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f799c13f650 0x7f799c1d75f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:41555/0 (socket says 172.21.15.155:41555) 2024-03-29T11:36:25.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.047+0000 7f79927fc640 1 -- 172.21.15.155:0/3455058178 learned_addr learned my addr 172.21.15.155:0/3455058178 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:25.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.047+0000 7f7993fff640 10 client.?.objecter ms_handle_connect 0x7f799c13f650 2024-03-29T11:36:25.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.047+0000 7f7993fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.047+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412393811 0 0) 0x7f799c0afed0 con 0x7f799c13f650 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f797c004eb0 con 0x7f799c13f650 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 10 client.?.objecter ms_handle_connect 0x7f799c0a9d70 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165793100 0 0) 0x7f799c0aed90 con 0x7f799c0a9d70 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.048+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f799c0afed0 con 0x7f799c0a9d70 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72692419 0 0) 0x7f799c0afed0 con 0x7f799c0a9d70 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f799c0aed90 con 0x7f799c0a9d70 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f798c001bc0 con 0x7f799c0a9d70 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 10 monclient(hunting): dump: 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:25.049 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:25.050 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:25.050 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:25.050 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:25.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:25.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 658727981 0 0) 0x7f799c0aed90 con 0x7f799c0a9d70 2024-03-29T11:36:25.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 >> v1:172.21.15.183:6789/0 conn(0x7f799c1457f0 legacy=0x7f799c0b0a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:36:25.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 >> v1:172.21.15.155:6789/0 conn(0x7f799c13f650 legacy=0x7f799c1d75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 1 monclient: found mon.c 2024-03-29T11:36:25.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c1d7d00 con 0x7f799c0a9d70 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f7993fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f79a14c1640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f79a14c1640 10 monclient: _renew_subs 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f79a14c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f79a14c1640 1 -- 172.21.15.155:0/3455058178 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799c0b1160 con 0x7f799c0a9d70 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f79a14c1640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f79a14c1640 10 monclient: _renew_subs 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f79a14c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f79a14c1640 1 -- 172.21.15.155:0/3455058178 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f799c0b04d0 con 0x7f799c0a9d70 2024-03-29T11:36:25.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.049+0000 7f79a14c1640 1 librados: init done 2024-03-29T11:36:25.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.050+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f798c004a90 con 0x7f799c0a9d70 2024-03-29T11:36:25.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.050+0000 7f7993fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.051+0000 7f7999ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:25.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.051+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f798c004da0 con 0x7f799c0a9d70 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.051+0000 7f7993fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.051+0000 7f7993fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.051+0000 7f7993fff640 10 monclient: dump: 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:25.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.051+0000 7f79a14c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.051+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f798c01ad70 con 0x7f799c0a9d70 2024-03-29T11:36:25.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.052+0000 7f79a14c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.052+0000 7f79a14c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:25.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.052+0000 7f79a14c1640 1 -- 172.21.15.155:0/3455058178 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7980005080 con 0x7f799c0a9d70 2024-03-29T11:36:25.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.052+0000 7f79927fc640 10 monclient: get_auth_request con 0x7f797c06dc90 auth_method 0 2024-03-29T11:36:25.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.052+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f798c084ae0 con 0x7f799c0a9d70 2024-03-29T11:36:25.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.052+0000 7f7993fff640 10 client.4292.objecter ms_dispatch 0x7f799c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:25.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.052+0000 7f7993fff640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:25.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.052+0000 7f7993fff640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:25.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.052+0000 7f7993fff640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T11:36:25.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.055+0000 7f7993fff640 10 client.4292.objecter ms_handle_connect 0x7f797c06dc90 2024-03-29T11:36:25.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.060+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f798c050480 con 0x7f799c0a9d70 2024-03-29T11:36:25.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.060+0000 7f7993fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.060+0000 7f7993fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.386+0000 7f79a14c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.386+0000 7f79a14c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.386+0000 7f79a14c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.386+0000 7f79a14c1640 1 -- 172.21.15.155:0/3455058178 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7980005330 con 0x7f799c0a9d70 2024-03-29T11:36:25.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.390+0000 7f7993fff640 1 -- 172.21.15.155:0/3455058178 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f798c0523b0 con 0x7f799c0a9d70 2024-03-29T11:36:25.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.390+0000 7f7993fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:25.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.390+0000 7f7993fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:25.389 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:36:25.390 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:25.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.395+0000 7f79a14c1640 10 librados: watch_flush enter 2024-03-29T11:36:25.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.395+0000 7f79a14c1640 10 librados: watch_flush exit 2024-03-29T11:36:25.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.395+0000 7f79a14c1640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T11:36:25.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.395+0000 7f79a14c1640 10 client.4292.objecter successfully canceled tick 2024-03-29T11:36:25.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.395+0000 7f79a14c1640 1 -- 172.21.15.155:0/3455058178 >> v1:172.21.15.155:6816/1760814708 conn(0x7f797c06dc90 legacy=0x7f797c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.395+0000 7f79a14c1640 10 monclient: shutdown 2024-03-29T11:36:25.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.395+0000 7f79a14c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.396+0000 7f79a14c1640 1 -- 172.21.15.155:0/3455058178 >> v1:172.21.15.155:6790/0 conn(0x7f799c0a9d70 legacy=0x7f799c0ae680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.396+0000 7f79a14c1640 1 -- 172.21.15.155:0/3455058178 shutdown_connections 2024-03-29T11:36:25.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.396+0000 7f79a14c1640 1 -- 172.21.15.155:0/3455058178 >> 172.21.15.155:0/3455058178 conn(0x7f799c066a00 msgr2=0x7f799c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.396+0000 7f79a14c1640 1 -- 172.21.15.155:0/3455058178 shutdown_connections 2024-03-29T11:36:25.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.396+0000 7f79a14c1640 1 -- 172.21.15.155:0/3455058178 wait complete. 2024-03-29T11:36:25.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:25.397+0000 7f79a14c1640 1 librados: shutdown 2024-03-29T11:36:25.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:32.220 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:32.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.379+0000 7fbe17f10640 10 monclient: get_monmap_and_config 2024-03-29T11:36:32.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.379+0000 7fbe17f10640 10 monclient: build_initial_monmap 2024-03-29T11:36:32.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.379+0000 7fbe17f10640 10 monclient: monmap: 2024-03-29T11:36:32.378 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:32.378 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:32.378 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:36:32.379467+0000 2024-03-29T11:36:32.378 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:36:32.379467+0000 2024-03-29T11:36:32.378 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:32.379 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:32.379 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:32.379 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:36:32.379 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:36:32.379 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:32.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.382+0000 7fbe17f10640 1 Processor -- start 2024-03-29T11:36:32.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.382+0000 7fbe17f10640 1 -- start start 2024-03-29T11:36:32.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.382+0000 7fbe17f10640 10 monclient: init 2024-03-29T11:36:32.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.383+0000 7fbe17f10640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:32.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.383+0000 7fbe17f10640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:32.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.383+0000 7fbe17f10640 10 monclient(hunting): picked mon.noname-c con 0x7fbe100a9ac0 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:32.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.383+0000 7fbe17f10640 10 monclient(hunting): picked mon.noname-a con 0x7fbe1013ee90 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:32.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.383+0000 7fbe17f10640 10 monclient(hunting): picked mon.noname-b con 0x7fbe100af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.383+0000 7fbe17f10640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe100afd10 con 0x7fbe1013ee90 2024-03-29T11:36:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.383+0000 7fbe17f10640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe100af6f0 con 0x7fbe100a9ac0 2024-03-29T11:36:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.383+0000 7fbe17f10640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe100aac80 con 0x7fbe100af900 2024-03-29T11:36:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.383+0000 7fbe17f10640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.383+0000 7fbe17f10640 10 monclient(hunting): authenticate will time out at 974.509558s 2024-03-29T11:36:32.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.389+0000 7fbe154ab640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fbe1013ee90 0x7fbe1013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:39599/0 (socket says 172.21.15.155:39599) 2024-03-29T11:36:32.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.389+0000 7fbe154ab640 1 -- 172.21.15.155:0/2277921453 learned_addr learned my addr 172.21.15.155:0/2277921453 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:32.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 1 -- 172.21.15.155:0/2277921453 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066615628 0 0) 0x7fbe100afd10 con 0x7fbe1013ee90 2024-03-29T11:36:32.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:32.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:32.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:32.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:36:32.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 1 -- 172.21.15.155:0/2277921453 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc003520 con 0x7fbe1013ee90 2024-03-29T11:36:32.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 1 -- 172.21.15.155:0/2277921453 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3957995621 0 0) 0x7fbdfc003520 con 0x7fbe1013ee90 2024-03-29T11:36:32.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 1 -- 172.21.15.155:0/2277921453 >> v1:172.21.15.155:6790/0 conn(0x7fbe100af900 legacy=0x7fbe10149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:32.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 1 -- 172.21.15.155:0/2277921453 >> v1:172.21.15.183:6789/0 conn(0x7fbe100a9ac0 legacy=0x7fbe100adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:32.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 10 monclient: _finish_hunting 0 2024-03-29T11:36:32.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 1 monclient: found mon.noname-a 2024-03-29T11:36:32.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:32.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:36:32.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 1 -- 172.21.15.155:0/2277921453 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe100aaf60 con 0x7fbe1013ee90 2024-03-29T11:36:32.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 10 monclient: _finish_auth 0 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 10 monclient: _check_auth_tickets 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.390+0000 7fbe14caa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.391+0000 7fbe17f10640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.391+0000 7fbe17f10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.391+0000 7fbe14caa640 1 -- 172.21.15.155:0/2277921453 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fbe08001a10 con 0x7fbe1013ee90 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.391+0000 7fbe14caa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.391+0000 7fbe14caa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.391+0000 7fbe14caa640 10 monclient: dump: 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:32.393 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:32.394 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:32.394 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:32.394 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:32.394 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:32.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.391+0000 7fbe17f10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:32.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.391+0000 7fbe14caa640 1 -- 172.21.15.155:0/2277921453 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe08001ca0 con 0x7fbe1013ee90 2024-03-29T11:36:32.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.391+0000 7fbe14caa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:32.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.391+0000 7fbe17f10640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:32.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.391+0000 7fbe17f10640 4 set_mon_vals no callback set 2024-03-29T11:36:32.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.392+0000 7fbe17f10640 10 monclient: shutdown 2024-03-29T11:36:32.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.392+0000 7fbe17f10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:32.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.392+0000 7fbe17f10640 1 -- 172.21.15.155:0/2277921453 >> v1:172.21.15.155:6789/0 conn(0x7fbe1013ee90 legacy=0x7fbe1013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:32.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.392+0000 7fbe17f10640 1 -- 172.21.15.155:0/2277921453 shutdown_connections 2024-03-29T11:36:32.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.392+0000 7fbe17f10640 1 -- 172.21.15.155:0/2277921453 >> 172.21.15.155:0/2277921453 conn(0x7fbe100b35e0 msgr2=0x7fbe100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:32.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.393+0000 7fbe17f10640 1 -- 172.21.15.155:0/2277921453 shutdown_connections 2024-03-29T11:36:32.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.394+0000 7fbe17f10640 1 -- 172.21.15.155:0/2277921453 wait complete. 2024-03-29T11:36:32.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.395+0000 7fbe17f10640 10 monclient: build_initial_monmap 2024-03-29T11:36:32.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.395+0000 7fbe17f10640 10 monclient: monmap: 2024-03-29T11:36:32.397 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.395+0000 7fbe17f10640 1 librados: starting msgr at 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.395+0000 7fbe17f10640 1 librados: starting objecter 2024-03-29T11:36:32.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.395+0000 7fbe17f10640 1 Processor -- start 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.395+0000 7fbe17f10640 1 -- start start 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.395+0000 7fbe17f10640 1 librados: setting wanted keys 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.395+0000 7fbe17f10640 1 librados: calling monclient init 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.395+0000 7fbe17f10640 10 monclient: init 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.396+0000 7fbe17f10640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.396+0000 7fbe17f10640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.396+0000 7fbe17f10640 10 monclient(hunting): picked mon.noname-a con 0x7fbe100a9ac0 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.396+0000 7fbe17f10640 10 monclient(hunting): picked mon.noname-c con 0x7fbe100af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.396+0000 7fbe17f10640 10 monclient(hunting): picked mon.noname-b con 0x7fbe1013ee90 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.396+0000 7fbe17f10640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe10070b90 con 0x7fbe100a9ac0 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.396+0000 7fbe17f10640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe10070490 con 0x7fbe1013ee90 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.396+0000 7fbe17f10640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe10070770 con 0x7fbe100af900 2024-03-29T11:36:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.396+0000 7fbe17f10640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.396+0000 7fbe17f10640 10 monclient(hunting): authenticate will time out at 974.522514s 2024-03-29T11:36:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.399+0000 7fbe15cac640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fbe100a9ac0 0x7fbe1006ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:39725/0 (socket says 172.21.15.155:39725) 2024-03-29T11:36:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.399+0000 7fbe15cac640 1 -- 172.21.15.155:0/4118514144 learned_addr learned my addr 172.21.15.155:0/4118514144 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.399+0000 7fbe067fc640 10 client.?.objecter ms_handle_connect 0x7fbe100a9ac0 2024-03-29T11:36:32.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.399+0000 7fbe067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.399+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920302399 0 0) 0x7fbe10070b90 con 0x7fbe100a9ac0 2024-03-29T11:36:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.399+0000 7fbe067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.399+0000 7fbe067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.399+0000 7fbe067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.399+0000 7fbe067fc640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:36:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.399+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf4004eb0 con 0x7fbe100a9ac0 2024-03-29T11:36:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.403+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2112318615 0 0) 0x7fbdf4004eb0 con 0x7fbe100a9ac0 2024-03-29T11:36:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.404+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe10070b90 con 0x7fbe100a9ac0 2024-03-29T11:36:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.404+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fbe00001870 con 0x7fbe100a9ac0 2024-03-29T11:36:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.404+0000 7fbe067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.404+0000 7fbe067fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.404+0000 7fbe067fc640 10 monclient(hunting): dump: 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.404+0000 7fbe067fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.404+0000 7fbe067fc640 10 client.?.objecter ms_handle_connect 0x7fbe1013ee90 2024-03-29T11:36:32.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.404+0000 7fbe067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.405+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183449052 0 0) 0x7fbe10070490 con 0x7fbe1013ee90 2024-03-29T11:36:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.405+0000 7fbe067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.405+0000 7fbe067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.405+0000 7fbe067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.405+0000 7fbe067fc640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:36:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.405+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf4005700 con 0x7fbe1013ee90 2024-03-29T11:36:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3128800254 0 0) 0x7fbe10070b90 con 0x7fbe100a9ac0 2024-03-29T11:36:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 >> v1:172.21.15.155:6790/0 conn(0x7fbe100af900 legacy=0x7fbe10071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 >> v1:172.21.15.183:6789/0 conn(0x7fbe1013ee90 legacy=0x7fbe10071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 1 monclient: found mon.a 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe10142180 con 0x7fbe100a9ac0 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe17f10640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe17f10640 10 monclient: _renew_subs 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe17f10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe17f10640 1 -- 172.21.15.155:0/4118514144 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe10142450 con 0x7fbe100a9ac0 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe17f10640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe17f10640 10 monclient: _renew_subs 2024-03-29T11:36:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe17f10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe17f10640 1 -- 172.21.15.155:0/4118514144 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe101e7540 con 0x7fbe100a9ac0 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe17f10640 1 librados: init done 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe00004900 con 0x7fbe100a9ac0 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fbe00001870 con 0x7fbe100a9ac0 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.406+0000 7fbe067fc640 10 monclient: dump: 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:32.411 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:32.412 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:32.412 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:32.412 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.408+0000 7fbe17f10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.408+0000 7fbe07fff640 4 set_mon_vals no callback set 2024-03-29T11:36:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.408+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fbe0001a8e0 con 0x7fbe100a9ac0 2024-03-29T11:36:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.409+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fbe00054470 con 0x7fbe100a9ac0 2024-03-29T11:36:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.409+0000 7fbe067fc640 10 client.4302.objecter ms_dispatch 0x7fbe10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.409+0000 7fbe067fc640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.409+0000 7fbe067fc640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.409+0000 7fbe067fc640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T11:36:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.411+0000 7fbe17f10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.411+0000 7fbe17f10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.412+0000 7fbe17f10640 1 -- 172.21.15.155:0/4118514144 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe10075f10 con 0x7fbe100a9ac0 2024-03-29T11:36:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.412+0000 7fbe154ab640 10 monclient: get_auth_request con 0x7fbdf406e220 auth_method 0 2024-03-29T11:36:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.413+0000 7fbe067fc640 10 client.4302.objecter ms_handle_connect 0x7fbdf406e220 2024-03-29T11:36:32.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.420+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe00050710 con 0x7fbe100a9ac0 2024-03-29T11:36:32.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.420+0000 7fbe067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:32.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.420+0000 7fbe067fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:32.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.746+0000 7fbe17f10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:32.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.746+0000 7fbe17f10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:32.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.746+0000 7fbe17f10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:32.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.746+0000 7fbe17f10640 1 -- 172.21.15.155:0/4118514144 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe100ad5f0 con 0x7fbe100a9ac0 2024-03-29T11:36:32.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.747+0000 7fbe067fc640 1 -- 172.21.15.155:0/4118514144 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fbe000508b0 con 0x7fbe100a9ac0 2024-03-29T11:36:32.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.747+0000 7fbe067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:32.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.747+0000 7fbe067fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:32.746 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:36:32.746 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.751+0000 7fbe17f10640 10 librados: watch_flush enter 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.751+0000 7fbe17f10640 10 librados: watch_flush exit 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.751+0000 7fbe17f10640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.751+0000 7fbe17f10640 10 client.4302.objecter successfully canceled tick 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.751+0000 7fbe17f10640 1 -- 172.21.15.155:0/4118514144 >> v1:172.21.15.155:6816/1760814708 conn(0x7fbdf406e220 legacy=0x7fbdf40706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.751+0000 7fbe17f10640 10 monclient: shutdown 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.751+0000 7fbe17f10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.752+0000 7fbe17f10640 1 -- 172.21.15.155:0/4118514144 >> v1:172.21.15.155:6789/0 conn(0x7fbe100a9ac0 legacy=0x7fbe1006ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.752+0000 7fbe17f10640 1 -- 172.21.15.155:0/4118514144 shutdown_connections 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.752+0000 7fbe17f10640 1 -- 172.21.15.155:0/4118514144 >> 172.21.15.155:0/4118514144 conn(0x7fbe100b35e0 msgr2=0x7fbe1013b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.752+0000 7fbe17f10640 1 -- 172.21.15.155:0/4118514144 shutdown_connections 2024-03-29T11:36:32.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.752+0000 7fbe17f10640 1 -- 172.21.15.155:0/4118514144 wait complete. 2024-03-29T11:36:32.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:32.752+0000 7fbe17f10640 1 librados: shutdown 2024-03-29T11:36:32.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:39.578 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:39.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.726+0000 7febf18e9640 10 monclient: get_monmap_and_config 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.726+0000 7febf18e9640 10 monclient: build_initial_monmap 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.726+0000 7febf18e9640 10 monclient: monmap: 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:36:39.727519+0000 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:36:39.727519+0000 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:36:39.726 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:39.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.730+0000 7febf18e9640 1 Processor -- start 2024-03-29T11:36:39.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.731+0000 7febf18e9640 1 -- start start 2024-03-29T11:36:39.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.731+0000 7febf18e9640 10 monclient: init 2024-03-29T11:36:39.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.732+0000 7febf18e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:39.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.732+0000 7febf18e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:39.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.732+0000 7febf18e9640 10 monclient(hunting): picked mon.noname-b con 0x7febe4006890 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:39.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.732+0000 7febf18e9640 10 monclient(hunting): picked mon.noname-c con 0x7febe4006c70 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:39.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.732+0000 7febf18e9640 10 monclient(hunting): picked mon.noname-a con 0x7febe402b4c0 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:39.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.732+0000 7febf18e9640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe402c830 con 0x7febe402b4c0 2024-03-29T11:36:39.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.732+0000 7febf18e9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe402afa0 con 0x7febe4006c70 2024-03-29T11:36:39.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.732+0000 7febf18e9640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe402aba0 con 0x7febe4006890 2024-03-29T11:36:39.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.732+0000 7febf18e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:39.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.732+0000 7febf18e9640 10 monclient(hunting): authenticate will time out at 981.859221s 2024-03-29T11:36:39.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.737+0000 7febebfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7febe4006c70 0x7febe40c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:38871/0 (socket says 172.21.15.155:38871) 2024-03-29T11:36:39.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.737+0000 7febebfff640 1 -- 172.21.15.155:0/2758455268 learned_addr learned my addr 172.21.15.155:0/2758455268 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:39.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642031478 0 0) 0x7febe402aba0 con 0x7febe4006890 2024-03-29T11:36:39.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:39.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:39.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc003520 con 0x7febe4006890 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836800895 0 0) 0x7febe402afa0 con 0x7febe4006c70 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe402aba0 con 0x7febe4006c70 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015331440 0 0) 0x7febe402c830 con 0x7febe402b4c0 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:39.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe402afa0 con 0x7febe402b4c0 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3701924348 0 0) 0x7febdc003520 con 0x7febe4006890 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 >> v1:172.21.15.183:6789/0 conn(0x7febe4006c70 legacy=0x7febe40c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 >> v1:172.21.15.155:6789/0 conn(0x7febe402b4c0 legacy=0x7febe40cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe402ad30 con 0x7febe4006890 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:39.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7febe0001690 con 0x7febe4006890 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febf18e9640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febf18e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febeb7fe640 10 monclient: dump: 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:39.743 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.739+0000 7febf18e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.740+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febe00017f0 con 0x7febe4006890 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.740+0000 7febeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.740+0000 7febeb7fe640 1 -- 172.21.15.155:0/2758455268 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7febe0001bc0 con 0x7febe4006890 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.740+0000 7febeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.740+0000 7febeb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.740+0000 7febeb7fe640 10 monclient: dump: 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:39.744 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:39.745 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:39.745 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:39.745 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:39.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.740+0000 7febf18e9640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:39.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.740+0000 7febf18e9640 4 set_mon_vals no callback set 2024-03-29T11:36:39.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.741+0000 7febf18e9640 10 monclient: shutdown 2024-03-29T11:36:39.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.741+0000 7febf18e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:39.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.741+0000 7febf18e9640 1 -- 172.21.15.155:0/2758455268 >> v1:172.21.15.155:6790/0 conn(0x7febe4006890 legacy=0x7febe40c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.745+0000 7febf18e9640 1 -- 172.21.15.155:0/2758455268 shutdown_connections 2024-03-29T11:36:39.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.745+0000 7febf18e9640 1 -- 172.21.15.155:0/2758455268 >> 172.21.15.155:0/2758455268 conn(0x7febe40058f0 msgr2=0x7febe4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:39.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.746+0000 7febf18e9640 1 -- 172.21.15.155:0/2758455268 shutdown_connections 2024-03-29T11:36:39.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.746+0000 7febf18e9640 1 -- 172.21.15.155:0/2758455268 wait complete. 2024-03-29T11:36:39.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.747+0000 7febf18e9640 10 monclient: build_initial_monmap 2024-03-29T11:36:39.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.747+0000 7febf18e9640 10 monclient: monmap: 2024-03-29T11:36:39.747 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:39.747 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:39.747 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:36:39.747 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:36:39.747 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:39.747 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:39.747 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:39.747 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:36:39.747 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:36:39.748 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:39.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.748+0000 7febf18e9640 1 librados: starting msgr at 2024-03-29T11:36:39.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.748+0000 7febf18e9640 1 librados: starting objecter 2024-03-29T11:36:39.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.748+0000 7febf18e9640 1 Processor -- start 2024-03-29T11:36:39.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.751+0000 7febf18e9640 1 -- start start 2024-03-29T11:36:39.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.751+0000 7febf18e9640 1 librados: setting wanted keys 2024-03-29T11:36:39.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.752+0000 7febf18e9640 1 librados: calling monclient init 2024-03-29T11:36:39.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.752+0000 7febf18e9640 10 monclient: init 2024-03-29T11:36:39.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.753+0000 7febf18e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:39.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.753+0000 7febf18e9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:39.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.753+0000 7febf18e9640 10 monclient(hunting): picked mon.noname-b con 0x7febe4006890 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:39.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.753+0000 7febf18e9640 10 monclient(hunting): picked mon.noname-c con 0x7febe402b4c0 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:39.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.754+0000 7febf18e9640 10 monclient(hunting): picked mon.noname-a con 0x7febe40c22b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:39.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.754+0000 7febf18e9640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe402d6e0 con 0x7febe40c22b0 2024-03-29T11:36:39.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.754+0000 7febf18e9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe40c4350 con 0x7febe4006890 2024-03-29T11:36:39.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.754+0000 7febf18e9640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe40c4630 con 0x7febe402b4c0 2024-03-29T11:36:39.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.754+0000 7febf18e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:39.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.754+0000 7febf18e9640 10 monclient(hunting): authenticate will time out at 981.881139s 2024-03-29T11:36:39.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.756+0000 7febf10e8640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7febe40c22b0 0x7febe416b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:43493/0 (socket says 172.21.15.155:43493) 2024-03-29T11:36:39.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.756+0000 7febf10e8640 1 -- 172.21.15.155:0/485477736 learned_addr learned my addr 172.21.15.155:0/485477736 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:39.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.757+0000 7febe97fa640 10 client.?.objecter ms_handle_connect 0x7febe40c22b0 2024-03-29T11:36:39.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.757+0000 7febe97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:39.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.757+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103509165 0 0) 0x7febe402d6e0 con 0x7febe40c22b0 2024-03-29T11:36:39.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.757+0000 7febe97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:39.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.757+0000 7febe97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:39.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.757+0000 7febe97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:39.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.757+0000 7febe97fa640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:36:39.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.757+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd0004eb0 con 0x7febe40c22b0 2024-03-29T11:36:39.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.759+0000 7febe97fa640 10 client.?.objecter ms_handle_connect 0x7febe4006890 2024-03-29T11:36:39.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.759+0000 7febe97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:39.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.759+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1364606718 0 0) 0x7febd0004eb0 con 0x7febe40c22b0 2024-03-29T11:36:39.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febe402d6e0 con 0x7febe40c22b0 2024-03-29T11:36:39.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7febec04fdd0 con 0x7febe40c22b0 2024-03-29T11:36:39.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:39.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:39.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 10 monclient(hunting): dump: 2024-03-29T11:36:39.760 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:39.760 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:39.760 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:39.760 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:39.760 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:39.760 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:39.760 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:39.760 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:39.760 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:39.760 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:39.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:39.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3195845125 0 0) 0x7febe402d6e0 con 0x7febe40c22b0 2024-03-29T11:36:39.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 >> v1:172.21.15.155:6790/0 conn(0x7febe402b4c0 legacy=0x7febe402cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 >> v1:172.21.15.183:6789/0 conn(0x7febe4006890 legacy=0x7febe40c2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 1 monclient: found mon.a 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe40c3700 con 0x7febe40c22b0 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 10 monclient: _finish_auth 0 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 10 client.?.objecter ms_handle_connect 0x7febe402b4c0 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febe97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febf18e9640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febf18e9640 10 monclient: _renew_subs 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febf18e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febf18e9640 1 -- 172.21.15.155:0/485477736 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe40bf030 con 0x7febe40c22b0 2024-03-29T11:36:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febf18e9640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febf18e9640 10 monclient: _renew_subs 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febf18e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febf18e9640 1 -- 172.21.15.155:0/485477736 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febe40bf4c0 con 0x7febe40c22b0 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.760+0000 7febf18e9640 1 librados: init done 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.762+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febec04f400 con 0x7febe40c22b0 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.762+0000 7febe97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.762+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7febec075710 con 0x7febe40c22b0 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.762+0000 7febe97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.762+0000 7febe97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.762+0000 7febeaffd640 4 set_mon_vals no callback set 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.762+0000 7febe97fa640 10 monclient: dump: 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:39.766 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.762+0000 7febd6ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.763+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7febec067bf0 con 0x7febe40c22b0 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.763+0000 7febd6ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.763+0000 7febd6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.763+0000 7febd6ffd640 1 -- 172.21.15.155:0/485477736 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febcc005080 con 0x7febe40c22b0 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.764+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7febec0ff5c0 con 0x7febe40c22b0 2024-03-29T11:36:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.764+0000 7febe97fa640 10 client.4308.objecter ms_dispatch 0x7febec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.764+0000 7febe97fa640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.764+0000 7febe97fa640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.764+0000 7febe97fa640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-29T11:36:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.764+0000 7febf08e7640 10 monclient: get_auth_request con 0x7febd006da10 auth_method 0 2024-03-29T11:36:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.765+0000 7febe97fa640 10 client.4308.objecter ms_handle_connect 0x7febd006da10 2024-03-29T11:36:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.777+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febec0cb0f0 con 0x7febe40c22b0 2024-03-29T11:36:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.777+0000 7febe97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:39.777+0000 7febe97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:40.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.101+0000 7febd6ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:40.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.101+0000 7febd6ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:40.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.101+0000 7febd6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:36:40.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.101+0000 7febd6ffd640 1 -- 172.21.15.155:0/485477736 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febcc005330 con 0x7febe40c22b0 2024-03-29T11:36:40.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.102+0000 7febe97fa640 1 -- 172.21.15.155:0/485477736 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7febec0ccf10 con 0x7febe40c22b0 2024-03-29T11:36:40.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.102+0000 7febe97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:40.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.102+0000 7febe97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:40.105 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:36:40.105 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:40.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.106+0000 7febd6ffd640 10 librados: watch_flush enter 2024-03-29T11:36:40.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.106+0000 7febd6ffd640 10 librados: watch_flush exit 2024-03-29T11:36:40.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.106+0000 7febd6ffd640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-29T11:36:40.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.106+0000 7febd6ffd640 10 client.4308.objecter successfully canceled tick 2024-03-29T11:36:40.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.107+0000 7febd6ffd640 1 -- 172.21.15.155:0/485477736 >> v1:172.21.15.155:6816/1760814708 conn(0x7febd006da10 legacy=0x7febd006fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.107+0000 7febd6ffd640 10 monclient: shutdown 2024-03-29T11:36:40.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.107+0000 7febd6ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:40.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.107+0000 7febd6ffd640 1 -- 172.21.15.155:0/485477736 >> v1:172.21.15.155:6789/0 conn(0x7febe40c22b0 legacy=0x7febe416b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:40.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.107+0000 7febd6ffd640 1 -- 172.21.15.155:0/485477736 shutdown_connections 2024-03-29T11:36:40.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.107+0000 7febd6ffd640 1 -- 172.21.15.155:0/485477736 >> 172.21.15.155:0/485477736 conn(0x7febe40058f0 msgr2=0x7febe4038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:40.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.107+0000 7febd6ffd640 1 -- 172.21.15.155:0/485477736 shutdown_connections 2024-03-29T11:36:40.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.107+0000 7febd6ffd640 1 -- 172.21.15.155:0/485477736 wait complete. 2024-03-29T11:36:40.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:40.107+0000 7febd6ffd640 1 librados: shutdown 2024-03-29T11:36:40.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:46.935 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:47.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.113+0000 7f7b1d94d640 10 monclient: get_monmap_and_config 2024-03-29T11:36:47.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.113+0000 7f7b1d94d640 10 monclient: build_initial_monmap 2024-03-29T11:36:47.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.113+0000 7f7b1d94d640 10 monclient: monmap: 2024-03-29T11:36:47.113 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:47.113 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:47.113 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:36:47.114843+0000 2024-03-29T11:36:47.113 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:36:47.114843+0000 2024-03-29T11:36:47.113 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:47.113 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:47.113 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:47.114 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:36:47.114 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:36:47.114 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:47.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.114+0000 7f7b1d94d640 1 Processor -- start 2024-03-29T11:36:47.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.114+0000 7f7b1d94d640 1 -- start start 2024-03-29T11:36:47.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.114+0000 7f7b1d94d640 10 monclient: init 2024-03-29T11:36:47.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.115+0000 7f7b1d94d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:47.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.115+0000 7f7b1d94d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:47.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.115+0000 7f7b1d94d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b181457f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:47.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b1d94d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b1813f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:47.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b1d94d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b180a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:47.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b1d94d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b180aaef0 con 0x7f7b181457f0 2024-03-29T11:36:47.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b1d94d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b18145bd0 con 0x7f7b180a9d70 2024-03-29T11:36:47.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b1d94d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b18145eb0 con 0x7f7b1813f650 2024-03-29T11:36:47.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b1d94d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:47.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b1d94d640 10 monclient(hunting): authenticate will time out at 989.243001s 2024-03-29T11:36:47.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b177fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f7b180a9d70 0x7f7b1814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:50000/0 (socket says 172.21.15.155:50000) 2024-03-29T11:36:47.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b177fe640 1 -- 172.21.15.155:0/1288105325 learned_addr learned my addr 172.21.15.155:0/1288105325 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:47.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756730058 0 0) 0x7f7b180aaef0 con 0x7f7b181457f0 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b15ffb640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.116+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b00003520 con 0x7f7b181457f0 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748844541 0 0) 0x7f7b18145bd0 con 0x7f7b180a9d70 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b00003840 con 0x7f7b180a9d70 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181760713 0 0) 0x7f7b18145eb0 con 0x7f7b1813f650 2024-03-29T11:36:47.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b00003b80 con 0x7f7b1813f650 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 354492859 0 0) 0x7f7b00003840 con 0x7f7b180a9d70 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 >> v1:172.21.15.155:6790/0 conn(0x7f7b1813f650 legacy=0x7f7b180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 >> v1:172.21.15.155:6789/0 conn(0x7f7b181457f0 legacy=0x7f7b1813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 1 monclient: found mon.noname-c 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b18146340 con 0x7f7b180a9d70 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:47.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7b080015f0 con 0x7f7b180a9d70 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient: dump: 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:47.121 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b1d94d640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b1d94d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b08001750 con 0x7f7b180a9d70 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b1d94d640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.117+0000 7f7b1d94d640 4 set_mon_vals no callback set 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.118+0000 7f7b15ffb640 1 -- 172.21.15.155:0/1288105325 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7b08001ba0 con 0x7f7b180a9d70 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.118+0000 7f7b15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.118+0000 7f7b15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.118+0000 7f7b15ffb640 10 monclient: dump: 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:47.122 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:47.123 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:47.123 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:47.123 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:47.123 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:47.123 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:47.123 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:47.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.118+0000 7f7b1d94d640 10 monclient: shutdown 2024-03-29T11:36:47.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.118+0000 7f7b1d94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:47.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.118+0000 7f7b1d94d640 1 -- 172.21.15.155:0/1288105325 >> v1:172.21.15.183:6789/0 conn(0x7f7b180a9d70 legacy=0x7f7b1814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:47.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.118+0000 7f7b1d94d640 1 -- 172.21.15.155:0/1288105325 shutdown_connections 2024-03-29T11:36:47.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.119+0000 7f7b1d94d640 1 -- 172.21.15.155:0/1288105325 >> 172.21.15.155:0/1288105325 conn(0x7f7b18066a00 msgr2=0x7f7b18066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:47.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.119+0000 7f7b1d94d640 1 -- 172.21.15.155:0/1288105325 shutdown_connections 2024-03-29T11:36:47.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.119+0000 7f7b1d94d640 1 -- 172.21.15.155:0/1288105325 wait complete. 2024-03-29T11:36:47.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.120+0000 7f7b1d94d640 10 monclient: build_initial_monmap 2024-03-29T11:36:47.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.120+0000 7f7b1d94d640 10 monclient: monmap: 2024-03-29T11:36:47.126 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:47.126 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:47.126 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:36:47.126 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:36:47.126 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.120+0000 7f7b1d94d640 1 librados: starting msgr at 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.120+0000 7f7b1d94d640 1 librados: starting objecter 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.120+0000 7f7b1d94d640 1 Processor -- start 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.121+0000 7f7b1d94d640 1 -- start start 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.121+0000 7f7b1d94d640 1 librados: setting wanted keys 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.121+0000 7f7b1d94d640 1 librados: calling monclient init 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.121+0000 7f7b1d94d640 10 monclient: init 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b1d94d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b1d94d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b1d94d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b180a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b1d94d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b1813f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b1d94d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b181457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b1d94d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1806fae0 con 0x7f7b180a9d70 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b1d94d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1813b6b0 con 0x7f7b181457f0 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b1d94d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1806e9f0 con 0x7f7b1813f650 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b1d94d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b1d94d640 10 monclient(hunting): authenticate will time out at 989.249216s 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b16ffd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7b180a9d70 0x7f7b1806f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:37034/0 (socket says 172.21.15.155:37034) 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7b16ffd640 1 -- 172.21.15.155:0/643280815 learned_addr learned my addr 172.21.15.155:0/643280815 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7afffff640 10 client.?.objecter ms_handle_connect 0x7f7b180a9d70 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7afffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7afffff640 10 client.?.objecter ms_handle_connect 0x7f7b181457f0 2024-03-29T11:36:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.122+0000 7f7afffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283037377 0 0) 0x7f7b1806fae0 con 0x7f7b180a9d70 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b04004eb0 con 0x7f7b180a9d70 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944566686 0 0) 0x7f7b1813b6b0 con 0x7f7b181457f0 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b1806fae0 con 0x7f7b181457f0 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 client.?.objecter ms_handle_connect 0x7f7b1813f650 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 399751021 0 0) 0x7f7b1806fae0 con 0x7f7b181457f0 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b1813b6b0 con 0x7f7b181457f0 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7b08001ba0 con 0x7f7b181457f0 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): dump: 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:47.130 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.123+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191903274 0 0) 0x7f7b1806e9f0 con 0x7f7b1813f650 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b1806fae0 con 0x7f7b1813f650 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3668940133 0 0) 0x7f7b04004eb0 con 0x7f7b180a9d70 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b1806e9f0 con 0x7f7b180a9d70 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7b10001c40 con 0x7f7b180a9d70 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:47.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 10 monclient(hunting): dump: 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1739859119 0 0) 0x7f7b1813b6b0 con 0x7f7b181457f0 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 >> v1:172.21.15.155:6790/0 conn(0x7f7b1813f650 legacy=0x7f7b181eb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 >> v1:172.21.15.155:6789/0 conn(0x7f7b180a9d70 legacy=0x7f7b1806f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:47.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 1 monclient: found mon.b 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1806ecd0 con 0x7f7b181457f0 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 10 monclient: _finish_auth 0 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.124+0000 7f7afffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7b1d94d640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7b1d94d640 10 monclient: _renew_subs 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7b1d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7b1d94d640 1 -- 172.21.15.155:0/643280815 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1806efb0 con 0x7f7b181457f0 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7b1d94d640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7b1d94d640 10 monclient: _renew_subs 2024-03-29T11:36:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7b1d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7b1d94d640 1 -- 172.21.15.155:0/643280815 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b181424f0 con 0x7f7b181457f0 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7b1d94d640 1 librados: init done 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b08004a90 con 0x7f7b181457f0 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7afffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7b08004da0 con 0x7f7b181457f0 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7afffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7afffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7afffff640 10 monclient: dump: 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:47.134 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:47.135 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:47.135 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:47.135 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:47.135 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:47.135 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.125+0000 7f7b14ff9640 4 set_mon_vals no callback set 2024-03-29T11:36:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.126+0000 7f7b1d94d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.126+0000 7f7b1d94d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.126+0000 7f7b1d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:36:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.126+0000 7f7b1d94d640 1 -- 172.21.15.155:0/643280815 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af8005080 con 0x7f7b181457f0 2024-03-29T11:36:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.128+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f7b0801ad70 con 0x7f7b181457f0 2024-03-29T11:36:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.129+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f7b08084a90 con 0x7f7b181457f0 2024-03-29T11:36:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.129+0000 7f7afffff640 10 client.4312.objecter ms_dispatch 0x7f7b18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.129+0000 7f7afffff640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.129+0000 7f7afffff640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.129+0000 7f7afffff640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T11:36:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.129+0000 7f7b16ffd640 10 monclient: get_auth_request con 0x7f7b0406e000 auth_method 0 2024-03-29T11:36:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.130+0000 7f7afffff640 10 client.4312.objecter ms_handle_connect 0x7f7b0406e000 2024-03-29T11:36:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.136+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b08050370 con 0x7f7b181457f0 2024-03-29T11:36:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.136+0000 7f7afffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.136+0000 7f7afffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:47.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.567+0000 7f7b1d94d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:47.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.567+0000 7f7b1d94d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:47.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.567+0000 7f7b1d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:36:47.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.567+0000 7f7b1d94d640 1 -- 172.21.15.155:0/643280815 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7af8005330 con 0x7f7b181457f0 2024-03-29T11:36:47.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.569+0000 7f7afffff640 1 -- 172.21.15.155:0/643280815 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f7b080522a0 con 0x7f7b181457f0 2024-03-29T11:36:47.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.569+0000 7f7afffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:47.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.569+0000 7f7afffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:47.569 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:36:47.569 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:47.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.573+0000 7f7b1d94d640 10 librados: watch_flush enter 2024-03-29T11:36:47.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.573+0000 7f7b1d94d640 10 librados: watch_flush exit 2024-03-29T11:36:47.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.573+0000 7f7b1d94d640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T11:36:47.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.573+0000 7f7b1d94d640 10 client.4312.objecter successfully canceled tick 2024-03-29T11:36:47.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.573+0000 7f7b1d94d640 1 -- 172.21.15.155:0/643280815 >> v1:172.21.15.155:6816/1760814708 conn(0x7f7b0406e000 legacy=0x7f7b040704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:47.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.573+0000 7f7b1d94d640 10 monclient: shutdown 2024-03-29T11:36:47.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.573+0000 7f7b1d94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:47.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.573+0000 7f7b1d94d640 1 -- 172.21.15.155:0/643280815 >> v1:172.21.15.183:6789/0 conn(0x7f7b181457f0 legacy=0x7f7b181eefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:47.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.573+0000 7f7b1d94d640 1 -- 172.21.15.155:0/643280815 shutdown_connections 2024-03-29T11:36:47.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.573+0000 7f7b1d94d640 1 -- 172.21.15.155:0/643280815 >> 172.21.15.155:0/643280815 conn(0x7f7b18066a00 msgr2=0x7f7b181d2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:47.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.574+0000 7f7b1d94d640 1 -- 172.21.15.155:0/643280815 shutdown_connections 2024-03-29T11:36:47.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.574+0000 7f7b1d94d640 1 -- 172.21.15.155:0/643280815 wait complete. 2024-03-29T11:36:47.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:47.574+0000 7f7b1d94d640 1 librados: shutdown 2024-03-29T11:36:47.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:54.396 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:54.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.541+0000 7fd754c1f640 10 monclient: get_monmap_and_config 2024-03-29T11:36:54.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.541+0000 7fd754c1f640 10 monclient: build_initial_monmap 2024-03-29T11:36:54.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.541+0000 7fd754c1f640 10 monclient: monmap: 2024-03-29T11:36:54.542 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:54.542 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:54.543 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:36:54.542941+0000 2024-03-29T11:36:54.543 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:36:54.542941+0000 2024-03-29T11:36:54.543 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:54.543 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:54.543 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:54.543 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:36:54.544 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:36:54.544 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:54.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.542+0000 7fd754c1f640 1 Processor -- start 2024-03-29T11:36:54.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.542+0000 7fd754c1f640 1 -- start start 2024-03-29T11:36:54.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.542+0000 7fd754c1f640 10 monclient: init 2024-03-29T11:36:54.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.543+0000 7fd754c1f640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:54.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.543+0000 7fd754c1f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:54.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.543+0000 7fd754c1f640 10 monclient(hunting): picked mon.noname-b con 0x7fd7500b2800 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:54.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.543+0000 7fd754c1f640 10 monclient(hunting): picked mon.noname-c con 0x7fd750142b20 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:54.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.543+0000 7fd754c1f640 10 monclient(hunting): picked mon.noname-a con 0x7fd75013c040 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:54.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.543+0000 7fd754c1f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7500aad10 con 0x7fd75013c040 2024-03-29T11:36:54.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.543+0000 7fd754c1f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd750141f30 con 0x7fd750142b20 2024-03-29T11:36:54.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.543+0000 7fd754c1f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7500aaf60 con 0x7fd7500b2800 2024-03-29T11:36:54.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.543+0000 7fd754c1f640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:54.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.543+0000 7fd754c1f640 10 monclient(hunting): authenticate will time out at 996.670797s 2024-03-29T11:36:54.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74effd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd750142b20 0x7fd75013b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:37630/0 (socket says 172.21.15.155:37630) 2024-03-29T11:36:54.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74ffff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd75013c040 0x7fd75014d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:54490/0 (socket says 172.21.15.155:54490) 2024-03-29T11:36:54.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74effd640 1 -- 172.21.15.155:0/2984320710 learned_addr learned my addr 172.21.15.155:0/2984320710 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:54.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341866906 0 0) 0x7fd750141f30 con 0x7fd750142b20 2024-03-29T11:36:54.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:36:54.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd738003520 con 0x7fd750142b20 2024-03-29T11:36:54.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488217778 0 0) 0x7fd7500aaf60 con 0x7fd7500b2800 2024-03-29T11:36:54.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:36:54.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.544+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd750141f30 con 0x7fd7500b2800 2024-03-29T11:36:54.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74f7fe640 1 -- 172.21.15.155:0/2984320710 >> v1:172.21.15.155:6790/0 conn(0x7fd7500b2800 legacy=0x7fd750142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:36:54.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74f7fe640 1 -- 172.21.15.155:0/2984320710 >> v1:172.21.15.155:6790/0 conn(0x7fd7500b2800 legacy=0x7fd750142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:36:54.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74f7fe640 1 --1- 172.21.15.155:0/2984320710 >> v1:172.21.15.155:6790/0 conn(0x7fd7500b2800 0x7fd750142410 :-1 s=OPENED pgs=237724 cs=1 l=1).handle_message read tag failed 2024-03-29T11:36:54.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74f7fe640 1 --1- 172.21.15.155:0/2984320710 >> v1:172.21.15.155:6790/0 conn(0x7fd7500b2800 0x7fd750142410 :-1 s=OPENED pgs=237724 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:36:54.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.155:6790/0 2024-03-29T11:36:54.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911814486 0 0) 0x7fd7500aad10 con 0x7fd75013c040 2024-03-29T11:36:54.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:36:54.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7500aaf60 con 0x7fd75013c040 2024-03-29T11:36:54.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822306080 0 0) 0x7fd7500aaf60 con 0x7fd75013c040 2024-03-29T11:36:54.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 >> v1:172.21.15.155:6790/0 conn(0x7fd7500b2800 legacy=0x7fd750142410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:36:54.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 >> v1:172.21.15.183:6789/0 conn(0x7fd750142b20 legacy=0x7fd75013b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:36:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:36:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7500b3900 con 0x7fd75013c040 2024-03-29T11:36:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:54.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:54.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.545+0000 7fd74e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:54.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd754c1f640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:36:54.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd754c1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:54.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd7400015b0 con 0x7fd75013c040 2024-03-29T11:36:54.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd74e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:54.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd74e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:54.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd74e7fc640 10 monclient: dump: 2024-03-29T11:36:54.580 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:54.580 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:54.580 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.580 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.580 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:54.581 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:54.581 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:54.581 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:54.581 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:54.581 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:54.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd754c1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:54.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd740001780 con 0x7fd75013c040 2024-03-29T11:36:54.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd74e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:54.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd754c1f640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:54.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd754c1f640 4 set_mon_vals no callback set 2024-03-29T11:36:54.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd74e7fc640 1 -- 172.21.15.155:0/2984320710 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd740001b50 con 0x7fd75013c040 2024-03-29T11:36:54.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd74e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:54.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd74e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:54.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.546+0000 7fd74e7fc640 10 monclient: dump: 2024-03-29T11:36:54.583 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:54.583 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:54.584 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.584 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.584 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:54.584 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:54.584 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:54.585 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:54.585 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:54.585 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:54.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.547+0000 7fd754c1f640 10 monclient: shutdown 2024-03-29T11:36:54.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.547+0000 7fd754c1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:54.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.547+0000 7fd754c1f640 1 -- 172.21.15.155:0/2984320710 >> v1:172.21.15.155:6789/0 conn(0x7fd75013c040 legacy=0x7fd75014d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.547+0000 7fd754c1f640 1 -- 172.21.15.155:0/2984320710 shutdown_connections 2024-03-29T11:36:54.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.547+0000 7fd754c1f640 1 -- 172.21.15.155:0/2984320710 >> 172.21.15.155:0/2984320710 conn(0x7fd75006d0e0 msgr2=0x7fd750066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:54.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.548+0000 7fd754c1f640 1 -- 172.21.15.155:0/2984320710 shutdown_connections 2024-03-29T11:36:54.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.548+0000 7fd754c1f640 1 -- 172.21.15.155:0/2984320710 wait complete. 2024-03-29T11:36:54.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.548+0000 7fd754c1f640 10 monclient: build_initial_monmap 2024-03-29T11:36:54.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.548+0000 7fd754c1f640 10 monclient: monmap: 2024-03-29T11:36:54.587 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:36:54.587 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:54.587 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:36:54.588 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:36:54.588 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:54.588 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:54.588 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:36:54.588 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:36:54.589 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:36:54.589 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:54.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.549+0000 7fd754c1f640 1 librados: starting msgr at 2024-03-29T11:36:54.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.549+0000 7fd754c1f640 1 librados: starting objecter 2024-03-29T11:36:54.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.549+0000 7fd754c1f640 1 Processor -- start 2024-03-29T11:36:54.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.549+0000 7fd754c1f640 1 -- start start 2024-03-29T11:36:54.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.549+0000 7fd754c1f640 1 librados: setting wanted keys 2024-03-29T11:36:54.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.549+0000 7fd754c1f640 1 librados: calling monclient init 2024-03-29T11:36:54.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.549+0000 7fd754c1f640 10 monclient: init 2024-03-29T11:36:54.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd754c1f640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:54.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd754c1f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:54.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd754c1f640 10 monclient(hunting): picked mon.noname-c con 0x7fd7500b2800 addr v1:172.21.15.155:6790/0 2024-03-29T11:36:54.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd754c1f640 10 monclient(hunting): picked mon.noname-b con 0x7fd75013c040 addr v1:172.21.15.183:6789/0 2024-03-29T11:36:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd754c1f640 10 monclient(hunting): picked mon.noname-a con 0x7fd750142b20 addr v1:172.21.15.155:6789/0 2024-03-29T11:36:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd754c1f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75013e8a0 con 0x7fd750142b20 2024-03-29T11:36:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd754c1f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75013e210 con 0x7fd75013c040 2024-03-29T11:36:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd754c1f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75013e480 con 0x7fd7500b2800 2024-03-29T11:36:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd754c1f640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:54.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd754c1f640 10 monclient(hunting): authenticate will time out at 996.677383s 2024-03-29T11:36:54.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd74ffff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd750142b20 0x7fd75013cdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:54870/0 (socket says 172.21.15.155:54870) 2024-03-29T11:36:54.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd74ffff640 1 -- 172.21.15.155:0/3415441813 learned_addr learned my addr 172.21.15.155:0/3415441813 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:36:54.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd74f7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fd7500b2800 0x7fd75013efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:42584/0 (socket says 172.21.15.155:42584) 2024-03-29T11:36:54.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd737fff640 10 client.?.objecter ms_handle_connect 0x7fd750142b20 2024-03-29T11:36:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd737fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd737fff640 10 client.?.objecter ms_handle_connect 0x7fd7500b2800 2024-03-29T11:36:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd737fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd737fff640 10 client.?.objecter ms_handle_connect 0x7fd75013c040 2024-03-29T11:36:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.550+0000 7fd737fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:54.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025219426 0 0) 0x7fd75013e480 con 0x7fd7500b2800 2024-03-29T11:36:54.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:36:54.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd73c004eb0 con 0x7fd7500b2800 2024-03-29T11:36:54.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120836160 0 0) 0x7fd75013e8a0 con 0x7fd750142b20 2024-03-29T11:36:54.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:36:54.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd75013e480 con 0x7fd750142b20 2024-03-29T11:36:54.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406753322 0 0) 0x7fd75013e210 con 0x7fd75013c040 2024-03-29T11:36:54.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:36:54.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd75013e8a0 con 0x7fd75013c040 2024-03-29T11:36:54.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2089308152 0 0) 0x7fd73c004eb0 con 0x7fd7500b2800 2024-03-29T11:36:54.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd75013e210 con 0x7fd7500b2800 2024-03-29T11:36:54.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd748001cf0 con 0x7fd7500b2800 2024-03-29T11:36:54.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:54.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:54.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): dump: 2024-03-29T11:36:54.599 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:54.600 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:54.600 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.600 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.600 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:54.600 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:54.601 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:54.601 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:54.601 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:54.601 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:54.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:54.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4293180379 0 0) 0x7fd75013e480 con 0x7fd750142b20 2024-03-29T11:36:54.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd73c004eb0 con 0x7fd750142b20 2024-03-29T11:36:54.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd740001a90 con 0x7fd750142b20 2024-03-29T11:36:54.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:54.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:54.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 10 monclient(hunting): dump: 2024-03-29T11:36:54.602 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:54.602 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:54.602 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.602 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.551+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1303373093 0 0) 0x7fd75013e8a0 con 0x7fd75013c040 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd75013e480 con 0x7fd75013c040 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd744001750 con 0x7fd75013c040 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 10 monclient(hunting): dump: 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:54.603 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1309801187 0 0) 0x7fd75013e210 con 0x7fd7500b2800 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 >> v1:172.21.15.183:6789/0 conn(0x7fd75013c040 legacy=0x7fd75013c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 >> v1:172.21.15.155:6789/0 conn(0x7fd750142b20 legacy=0x7fd75013cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 1 monclient: found mon.c 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:54.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7501e12a0 con 0x7fd7500b2800 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd754c1f640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd754c1f640 10 monclient: _renew_subs 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd754c1f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd754c1f640 1 -- 172.21.15.155:0/3415441813 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7501de270 con 0x7fd7500b2800 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd754c1f640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd754c1f640 10 monclient: _renew_subs 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd754c1f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd754c1f640 1 -- 172.21.15.155:0/3415441813 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7501de750 con 0x7fd7500b2800 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd754c1f640 1 librados: init done 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd748003990 con 0x7fd7500b2800 2024-03-29T11:36:54.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd748003d60 con 0x7fd7500b2800 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd737fff640 10 monclient: dump: 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:36:54.606 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.552+0000 7fd74dffb640 4 set_mon_vals no callback set 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.554+0000 7fd754c1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.554+0000 7fd754c1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.554+0000 7fd754c1f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.554+0000 7fd754c1f640 1 -- 172.21.15.155:0/3415441813 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd730005080 con 0x7fd7500b2800 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.555+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fd748003ec0 con 0x7fd7500b2800 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.556+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fd748084d80 con 0x7fd7500b2800 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.556+0000 7fd737fff640 10 client.4310.objecter ms_dispatch 0x7fd750000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.556+0000 7fd737fff640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.556+0000 7fd737fff640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.556+0000 7fd737fff640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.556+0000 7fd74effd640 10 monclient: get_auth_request con 0x7fd73c06e0e0 auth_method 0 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.556+0000 7fd737fff640 10 client.4310.objecter ms_handle_connect 0x7fd73c06e0e0 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.564+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd748050660 con 0x7fd7500b2800 2024-03-29T11:36:54.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.564+0000 7fd737fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:54.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.564+0000 7fd737fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:54.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.929+0000 7fd754c1f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:54.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.929+0000 7fd754c1f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:54.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.929+0000 7fd754c1f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:36:54.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.929+0000 7fd754c1f640 1 -- 172.21.15.155:0/3415441813 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd730005330 con 0x7fd7500b2800 2024-03-29T11:36:54.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.931+0000 7fd737fff640 1 -- 172.21.15.155:0/3415441813 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fd748052590 con 0x7fd7500b2800 2024-03-29T11:36:54.931 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:36:54.931 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:54.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.931+0000 7fd737fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:54.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.931+0000 7fd737fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:54.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 10 librados: watch_flush enter 2024-03-29T11:36:54.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 10 librados: watch_flush exit 2024-03-29T11:36:54.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T11:36:54.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 10 client.4310.objecter successfully canceled tick 2024-03-29T11:36:54.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 1 -- 172.21.15.155:0/3415441813 >> v1:172.21.15.155:6816/1760814708 conn(0x7fd73c06e0e0 legacy=0x7fd73c0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 10 monclient: shutdown 2024-03-29T11:36:54.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:54.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 1 -- 172.21.15.155:0/3415441813 >> v1:172.21.15.155:6790/0 conn(0x7fd7500b2800 legacy=0x7fd75013efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 1 -- 172.21.15.155:0/3415441813 shutdown_connections 2024-03-29T11:36:54.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 1 -- 172.21.15.155:0/3415441813 >> 172.21.15.155:0/3415441813 conn(0x7fd75006d0e0 msgr2=0x7fd750066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:54.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 1 -- 172.21.15.155:0/3415441813 shutdown_connections 2024-03-29T11:36:54.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 1 -- 172.21.15.155:0/3415441813 wait complete. 2024-03-29T11:36:54.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:36:54.935+0000 7fd754c1f640 1 librados: shutdown 2024-03-29T11:36:54.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:01.758 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:01.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.910+0000 7fa2ab245640 10 monclient: get_monmap_and_config 2024-03-29T11:37:01.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.910+0000 7fa2ab245640 10 monclient: build_initial_monmap 2024-03-29T11:37:01.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.910+0000 7fa2ab245640 10 monclient: monmap: 2024-03-29T11:37:01.911 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:01.911 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:01.911 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:37:01.911777+0000 2024-03-29T11:37:01.911 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:37:01.911777+0000 2024-03-29T11:37:01.911 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:01.911 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:01.911 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:01.911 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:37:01.911 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:37:01.911 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:01.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.911+0000 7fa2ab245640 1 Processor -- start 2024-03-29T11:37:01.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.911+0000 7fa2ab245640 1 -- start start 2024-03-29T11:37:01.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.911+0000 7fa2ab245640 10 monclient: init 2024-03-29T11:37:01.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.912+0000 7fa2ab245640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:01.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.912+0000 7fa2ab245640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:01.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.912+0000 7fa2ab245640 10 monclient(hunting): picked mon.noname-a con 0x7fa2a40b0770 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:01.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.912+0000 7fa2ab245640 10 monclient(hunting): picked mon.noname-b con 0x7fa2a414a700 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.912+0000 7fa2ab245640 10 monclient(hunting): picked mon.noname-c con 0x7fa2a40ac5a0 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.912+0000 7fa2ab245640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a40aba70 con 0x7fa2a40b0770 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.912+0000 7fa2ab245640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a40aecb0 con 0x7fa2a40ac5a0 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.912+0000 7fa2ab245640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a40af150 con 0x7fa2a414a700 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.912+0000 7fa2ab245640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.912+0000 7fa2ab245640 10 monclient(hunting): authenticate will time out at 1004.039314s 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.913+0000 7fa2a3fff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa2a414a700 0x7fa2a40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33988/0 (socket says 172.21.15.155:33988) 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.914+0000 7fa2a3fff640 1 -- 172.21.15.155:0/4150063270 learned_addr learned my addr 172.21.15.155:0/4150063270 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.914+0000 7fa2a37fe640 1 -- 172.21.15.155:0/4150063270 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518757219 0 0) 0x7fa2a40af150 con 0x7fa2a414a700 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.914+0000 7fa2a37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.914+0000 7fa2a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.914+0000 7fa2a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:01.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.914+0000 7fa2a37fe640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:37:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.914+0000 7fa2a37fe640 1 -- 172.21.15.155:0/4150063270 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa28c003520 con 0x7fa2a414a700 2024-03-29T11:37:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 1 -- 172.21.15.155:0/4150063270 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2660864276 0 0) 0x7fa28c003520 con 0x7fa2a414a700 2024-03-29T11:37:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 1 -- 172.21.15.155:0/4150063270 >> v1:172.21.15.183:6789/0 conn(0x7fa2a40ac5a0 legacy=0x7fa2a4151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:01.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 1 -- 172.21.15.155:0/4150063270 >> v1:172.21.15.155:6789/0 conn(0x7fa2a40b0770 legacy=0x7fa2a4149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:01.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:01.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 1 monclient: found mon.noname-b 2024-03-29T11:37:01.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 1 -- 172.21.15.155:0/4150063270 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a40ac9d0 con 0x7fa2a414a700 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 1 -- 172.21.15.155:0/4150063270 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa29c001a10 con 0x7fa2a414a700 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.915+0000 7fa2a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.916+0000 7fa2a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.916+0000 7fa2a37fe640 10 monclient: dump: 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:01.919 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.916+0000 7fa2a37fe640 1 -- 172.21.15.155:0/4150063270 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa29c001ca0 con 0x7fa2a414a700 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.916+0000 7fa2a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.916+0000 7fa2a37fe640 1 -- 172.21.15.155:0/4150063270 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa29c002030 con 0x7fa2a414a700 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.916+0000 7fa2a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.916+0000 7fa2a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.916+0000 7fa2a37fe640 10 monclient: dump: 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:01.920 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.916+0000 7fa2ab245640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.916+0000 7fa2ab245640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.916+0000 7fa2ab245640 4 set_mon_vals no callback set 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.917+0000 7fa2ab245640 10 monclient: shutdown 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.917+0000 7fa2ab245640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.917+0000 7fa2ab245640 1 -- 172.21.15.155:0/4150063270 >> v1:172.21.15.155:6790/0 conn(0x7fa2a414a700 legacy=0x7fa2a40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.918+0000 7fa2ab245640 1 -- 172.21.15.155:0/4150063270 shutdown_connections 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.918+0000 7fa2ab245640 1 -- 172.21.15.155:0/4150063270 >> 172.21.15.155:0/4150063270 conn(0x7fa2a406b660 msgr2=0x7fa2a406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:01.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.918+0000 7fa2ab245640 1 -- 172.21.15.155:0/4150063270 shutdown_connections 2024-03-29T11:37:01.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.918+0000 7fa2ab245640 1 -- 172.21.15.155:0/4150063270 wait complete. 2024-03-29T11:37:01.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.919+0000 7fa2ab245640 10 monclient: build_initial_monmap 2024-03-29T11:37:01.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.919+0000 7fa2ab245640 10 monclient: monmap: 2024-03-29T11:37:01.924 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:01.924 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:01.924 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:37:01.924 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:37:01.924 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:01.924 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:01.924 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:01.924 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:37:01.924 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.920+0000 7fa2ab245640 1 librados: starting msgr at 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.920+0000 7fa2ab245640 1 librados: starting objecter 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.920+0000 7fa2ab245640 1 Processor -- start 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.920+0000 7fa2ab245640 1 -- start start 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.920+0000 7fa2ab245640 1 librados: setting wanted keys 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.920+0000 7fa2ab245640 1 librados: calling monclient init 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.920+0000 7fa2ab245640 10 monclient: init 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.921+0000 7fa2ab245640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.921+0000 7fa2ab245640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.921+0000 7fa2ab245640 10 monclient(hunting): picked mon.noname-a con 0x7fa2a40ac5a0 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.921+0000 7fa2ab245640 10 monclient(hunting): picked mon.noname-b con 0x7fa2a40b0770 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.921+0000 7fa2ab245640 10 monclient(hunting): picked mon.noname-c con 0x7fa2a41df900 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:01.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.921+0000 7fa2ab245640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a41265f0 con 0x7fa2a40ac5a0 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.921+0000 7fa2ab245640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a4145b10 con 0x7fa2a40b0770 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.921+0000 7fa2ab245640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a41239d0 con 0x7fa2a41df900 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.921+0000 7fa2ab245640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.921+0000 7fa2ab245640 10 monclient(hunting): authenticate will time out at 1004.048349s 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a97e2640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa2a41df900 0x7fa2a4145400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:34136/0 (socket says 172.21.15.155:34136) 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a97e2640 1 -- 172.21.15.155:0/4180000963 learned_addr learned my addr 172.21.15.155:0/4180000963 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a17fa640 10 client.?.objecter ms_handle_connect 0x7fa2a41df900 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506424968 0 0) 0x7fa2a41239d0 con 0x7fa2a41df900 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a17fa640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:37:01.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa290004eb0 con 0x7fa2a41df900 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1205698976 0 0) 0x7fa290004eb0 con 0x7fa2a41df900 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.923+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2a41239d0 con 0x7fa2a41df900 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.924+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa2980017f0 con 0x7fa2a41df900 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.924+0000 7fa2a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.924+0000 7fa2a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.924+0000 7fa2a17fa640 10 monclient(hunting): dump: 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:01.927 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:01.928 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:01.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.924+0000 7fa2a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:01.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.924+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 364807337 0 0) 0x7fa2a41239d0 con 0x7fa2a41df900 2024-03-29T11:37:01.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.924+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 >> v1:172.21.15.183:6789/0 conn(0x7fa2a40b0770 legacy=0x7fa2a4144cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:01.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 >> v1:172.21.15.155:6789/0 conn(0x7fa2a40ac5a0 legacy=0x7fa2a4126a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:01.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2a17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:01.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2a17fa640 1 monclient: found mon.c 2024-03-29T11:37:01.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2a17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a4128610 con 0x7fa2a41df900 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2a17fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2a17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2ab245640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2ab245640 10 monclient: _renew_subs 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2ab245640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2ab245640 1 -- 172.21.15.155:0/4180000963 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a4125fb0 con 0x7fa2a41df900 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2ab245640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2ab245640 10 monclient: _renew_subs 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2ab245640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2ab245640 1 -- 172.21.15.155:0/4180000963 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a41df070 con 0x7fa2a41df900 2024-03-29T11:37:01.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2ab245640 1 librados: init done 2024-03-29T11:37:01.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2a17fa640 10 client.4316.objecter ms_handle_connect 0x7fa2a40ac5a0 2024-03-29T11:37:01.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.927+0000 7fa2a17fa640 10 client.4316.objecter resend_mon_ops 2024-03-29T11:37:01.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.929+0000 7fa2ab245640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:01.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.929+0000 7fa2ab245640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:01.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.929+0000 7fa2ab245640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:37:01.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.929+0000 7fa2ab245640 1 -- 172.21.15.155:0/4180000963 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a4075f10 con 0x7fa2a41df900 2024-03-29T11:37:01.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.930+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa298001950 con 0x7fa2a41df900 2024-03-29T11:37:01.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.930+0000 7fa2a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:01.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.930+0000 7fa2a27fc640 4 set_mon_vals no callback set 2024-03-29T11:37:01.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.932+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa298004aa0 con 0x7fa2a41df900 2024-03-29T11:37:01.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.932+0000 7fa2a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:01.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.932+0000 7fa2a17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:01.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.932+0000 7fa2a17fa640 10 monclient: dump: 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.932+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fa29801aaf0 con 0x7fa2a41df900 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.933+0000 7fa2a8fe1640 10 monclient: get_auth_request con 0x7fa29006db10 auth_method 0 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.933+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fa298084a70 con 0x7fa2a41df900 2024-03-29T11:37:01.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.933+0000 7fa2a17fa640 10 client.4316.objecter ms_dispatch 0x7fa2a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:01.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.933+0000 7fa2a17fa640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:01.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.933+0000 7fa2a17fa640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:01.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.933+0000 7fa2a17fa640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T11:37:01.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.933+0000 7fa2a17fa640 10 client.4316.objecter ms_handle_connect 0x7fa29006db10 2024-03-29T11:37:01.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.940+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa298050200 con 0x7fa2a41df900 2024-03-29T11:37:01.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.940+0000 7fa2a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:01.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:01.940+0000 7fa2a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:02.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.278+0000 7fa2ab245640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:02.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.278+0000 7fa2ab245640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:02.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.278+0000 7fa2ab245640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:37:02.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.278+0000 7fa2ab245640 1 -- 172.21.15.155:0/4180000963 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2a40b2950 con 0x7fa2a41df900 2024-03-29T11:37:02.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.279+0000 7fa2a17fa640 1 -- 172.21.15.155:0/4180000963 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fa298052300 con 0x7fa2a41df900 2024-03-29T11:37:02.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.279+0000 7fa2a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:02.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.279+0000 7fa2a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:02.280 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:37:02.280 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:02.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.283+0000 7fa2ab245640 10 librados: watch_flush enter 2024-03-29T11:37:02.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.284+0000 7fa2ab245640 10 librados: watch_flush exit 2024-03-29T11:37:02.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.284+0000 7fa2ab245640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T11:37:02.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.284+0000 7fa2ab245640 10 client.4316.objecter successfully canceled tick 2024-03-29T11:37:02.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.286+0000 7fa2ab245640 1 -- 172.21.15.155:0/4180000963 >> v1:172.21.15.155:6816/1760814708 conn(0x7fa29006db10 legacy=0x7fa29006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:02.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.286+0000 7fa2ab245640 10 monclient: shutdown 2024-03-29T11:37:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.286+0000 7fa2ab245640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.286+0000 7fa2ab245640 1 -- 172.21.15.155:0/4180000963 >> v1:172.21.15.155:6790/0 conn(0x7fa2a41df900 legacy=0x7fa2a4145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.286+0000 7fa2ab245640 1 -- 172.21.15.155:0/4180000963 shutdown_connections 2024-03-29T11:37:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.286+0000 7fa2ab245640 1 -- 172.21.15.155:0/4180000963 >> 172.21.15.155:0/4180000963 conn(0x7fa2a406b660 msgr2=0x7fa2a40b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.286+0000 7fa2ab245640 1 -- 172.21.15.155:0/4180000963 shutdown_connections 2024-03-29T11:37:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.287+0000 7fa2ab245640 1 -- 172.21.15.155:0/4180000963 wait complete. 2024-03-29T11:37:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:02.287+0000 7fa2ab245640 1 librados: shutdown 2024-03-29T11:37:02.306 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:09.113 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:09.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.256+0000 7fa652463640 10 monclient: get_monmap_and_config 2024-03-29T11:37:09.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.256+0000 7fa652463640 10 monclient: build_initial_monmap 2024-03-29T11:37:09.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.256+0000 7fa652463640 10 monclient: monmap: 2024-03-29T11:37:09.256 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:09.257 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:09.257 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:37:09.256614+0000 2024-03-29T11:37:09.257 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:37:09.256614+0000 2024-03-29T11:37:09.257 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:09.257 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:09.257 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:09.257 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:37:09.257 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:37:09.257 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:09.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.257+0000 7fa652463640 1 Processor -- start 2024-03-29T11:37:09.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.257+0000 7fa652463640 1 -- start start 2024-03-29T11:37:09.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.257+0000 7fa652463640 10 monclient: init 2024-03-29T11:37:09.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.258+0000 7fa652463640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:09.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.258+0000 7fa652463640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:09.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.258+0000 7fa652463640 10 monclient(hunting): picked mon.noname-c con 0x7fa64c0a9ac0 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:09.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.258+0000 7fa652463640 10 monclient(hunting): picked mon.noname-a con 0x7fa64c13ee90 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:09.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.258+0000 7fa650a00640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa64c0af900 0x7fa64c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55184/0 (socket says 172.21.15.155:55184) 2024-03-29T11:37:09.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.258+0000 7fa650a00640 1 -- 172.21.15.155:0/1416262918 learned_addr learned my addr 172.21.15.155:0/1416262918 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:09.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.258+0000 7fa652463640 10 monclient(hunting): picked mon.noname-b con 0x7fa64c0af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:09.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.259+0000 7fa652463640 1 -- 172.21.15.155:0/1416262918 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c0afd10 con 0x7fa64c13ee90 2024-03-29T11:37:09.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.259+0000 7fa652463640 1 -- 172.21.15.155:0/1416262918 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c0af6f0 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.259+0000 7fa652463640 1 -- 172.21.15.155:0/1416262918 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c0aac80 con 0x7fa64c0af900 2024-03-29T11:37:09.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.259+0000 7fa652463640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:09.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.259+0000 7fa652463640 10 monclient(hunting): authenticate will time out at 1011.385227s 2024-03-29T11:37:09.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.261+0000 7fa64affd640 1 -- 172.21.15.155:0/1416262918 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060388011 0 0) 0x7fa64c0aac80 con 0x7fa64c0af900 2024-03-29T11:37:09.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.261+0000 7fa64affd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:09.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.261+0000 7fa64affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:09.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.261+0000 7fa64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:09.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.261+0000 7fa64affd640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:37:09.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.261+0000 7fa64affd640 1 -- 172.21.15.155:0/1416262918 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa634003520 con 0x7fa64c0af900 2024-03-29T11:37:09.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.261+0000 7fa64affd640 1 -- 172.21.15.155:0/1416262918 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1634882370 0 0) 0x7fa634003520 con 0x7fa64c0af900 2024-03-29T11:37:09.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.261+0000 7fa64affd640 1 -- 172.21.15.155:0/1416262918 >> v1:172.21.15.183:6789/0 conn(0x7fa64c0a9ac0 legacy=0x7fa64c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:09.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.261+0000 7fa64affd640 1 -- 172.21.15.155:0/1416262918 >> v1:172.21.15.155:6789/0 conn(0x7fa64c13ee90 legacy=0x7fa64c13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:09.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:09.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 1 monclient: found mon.noname-b 2024-03-29T11:37:09.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 1 -- 172.21.15.155:0/1416262918 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0aaf60 con 0x7fa64c0af900 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 10 monclient: _finish_auth 0 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 1 -- 172.21.15.155:0/1416262918 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa644001a10 con 0x7fa64c0af900 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa64affd640 10 monclient: dump: 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:09.263 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:09.264 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:09.264 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:09.264 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:09.264 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:09.264 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:09.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa652463640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:37:09.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.263+0000 7fa652463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:09.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.264+0000 7fa64affd640 1 -- 172.21.15.155:0/1416262918 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa644001be0 con 0x7fa64c0af900 2024-03-29T11:37:09.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.264+0000 7fa64affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:09.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.264+0000 7fa64affd640 1 -- 172.21.15.155:0/1416262918 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa644002030 con 0x7fa64c0af900 2024-03-29T11:37:09.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.264+0000 7fa64affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:09.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.264+0000 7fa64affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:09.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.264+0000 7fa64affd640 10 monclient: dump: 2024-03-29T11:37:09.265 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:09.265 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:09.265 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:09.265 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:09.265 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.264+0000 7fa652463640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.264+0000 7fa652463640 4 set_mon_vals no callback set 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.265+0000 7fa652463640 10 monclient: shutdown 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.265+0000 7fa652463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.265+0000 7fa652463640 1 -- 172.21.15.155:0/1416262918 >> v1:172.21.15.155:6790/0 conn(0x7fa64c0af900 legacy=0x7fa64c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.265+0000 7fa652463640 1 -- 172.21.15.155:0/1416262918 shutdown_connections 2024-03-29T11:37:09.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.265+0000 7fa652463640 1 -- 172.21.15.155:0/1416262918 >> 172.21.15.155:0/1416262918 conn(0x7fa64c0b35e0 msgr2=0x7fa64c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:09.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.266+0000 7fa652463640 1 -- 172.21.15.155:0/1416262918 shutdown_connections 2024-03-29T11:37:09.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.268+0000 7fa652463640 1 -- 172.21.15.155:0/1416262918 wait complete. 2024-03-29T11:37:09.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.268+0000 7fa652463640 10 monclient: build_initial_monmap 2024-03-29T11:37:09.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.268+0000 7fa652463640 10 monclient: monmap: 2024-03-29T11:37:09.269 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:09.269 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:09.269 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:37:09.269 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:37:09.269 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:09.270 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:09.270 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:09.270 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:37:09.270 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:37:09.270 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:09.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.269+0000 7fa652463640 1 librados: starting msgr at 2024-03-29T11:37:09.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.269+0000 7fa652463640 1 librados: starting objecter 2024-03-29T11:37:09.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.269+0000 7fa652463640 1 Processor -- start 2024-03-29T11:37:09.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.269+0000 7fa652463640 1 -- start start 2024-03-29T11:37:09.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.269+0000 7fa652463640 1 librados: setting wanted keys 2024-03-29T11:37:09.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.269+0000 7fa652463640 1 librados: calling monclient init 2024-03-29T11:37:09.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.269+0000 7fa652463640 10 monclient: init 2024-03-29T11:37:09.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.270+0000 7fa652463640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:09.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.270+0000 7fa652463640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:09.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.270+0000 7fa652463640 10 monclient(hunting): picked mon.noname-a con 0x7fa64c0a9ac0 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:09.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.270+0000 7fa652463640 10 monclient(hunting): picked mon.noname-c con 0x7fa64c0af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:09.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.271+0000 7fa652463640 10 monclient(hunting): picked mon.noname-b con 0x7fa64c13ee90 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:09.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.271+0000 7fa652463640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c071070 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.271+0000 7fa652463640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c06ff20 con 0x7fa64c13ee90 2024-03-29T11:37:09.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.271+0000 7fa652463640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c0700b0 con 0x7fa64c0af900 2024-03-29T11:37:09.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.271+0000 7fa652463640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:09.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.271+0000 7fa652463640 10 monclient(hunting): authenticate will time out at 1011.397622s 2024-03-29T11:37:09.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.273+0000 7fa64b7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa64c0af900 0x7fa64c1e0ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:58086/0 (socket says 172.21.15.155:58086) 2024-03-29T11:37:09.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.273+0000 7fa64b7fe640 1 -- 172.21.15.155:0/2531949081 learned_addr learned my addr 172.21.15.155:0/2531949081 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:09.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.273+0000 7fa648ff9640 10 client.?.objecter ms_handle_connect 0x7fa64c0a9ac0 2024-03-29T11:37:09.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.273+0000 7fa648ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:09.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694184126 0 0) 0x7fa64c071070 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:09.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:09.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:09.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:37:09.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa638004eb0 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1872316516 0 0) 0x7fa638004eb0 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa638005300 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa640001510 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:09.275 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.274+0000 7fa648ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2589256265 0 0) 0x7fa638005300 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 >> v1:172.21.15.155:6790/0 conn(0x7fa64c0af900 legacy=0x7fa64c1e0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 >> v1:172.21.15.183:6789/0 conn(0x7fa64c13ee90 legacy=0x7fa64c1e3e30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa648ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa648ff9640 1 monclient: found mon.a 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa648ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa648ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c1e1700 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa648ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa648ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa648ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:09.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa652463640 5 monclient: authenticate success, global_id 4326 2024-03-29T11:37:09.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa652463640 10 monclient: _renew_subs 2024-03-29T11:37:09.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa652463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:09.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa652463640 1 -- 172.21.15.155:0/2531949081 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c1e18f0 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa652463640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:09.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa652463640 10 monclient: _renew_subs 2024-03-29T11:37:09.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa652463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:09.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa652463640 1 -- 172.21.15.155:0/2531949081 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c141980 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.275+0000 7fa652463640 1 librados: init done 2024-03-29T11:37:09.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.276+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa640001ae0 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.276+0000 7fa648ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:09.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.276+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa640001510 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.276+0000 7fa648ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.276+0000 7fa648ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.276+0000 7fa648ff9640 10 monclient: dump: 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:09.280 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.277+0000 7fa64a7fc640 4 set_mon_vals no callback set 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.277+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fa640019780 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.278+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fa640053200 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.278+0000 7fa648ff9640 10 client.4326.objecter ms_dispatch 0x7fa64c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.278+0000 7fa648ff9640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.278+0000 7fa648ff9640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.278+0000 7fa648ff9640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.278+0000 7fa652463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.278+0000 7fa652463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.278+0000 7fa652463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.278+0000 7fa652463640 1 -- 172.21.15.155:0/2531949081 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62c005080 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.278+0000 7fa64b7fe640 10 monclient: get_auth_request con 0x7fa63806e2a0 auth_method 0 2024-03-29T11:37:09.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.279+0000 7fa648ff9640 10 client.4326.objecter ms_handle_connect 0x7fa63806e2a0 2024-03-29T11:37:09.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.290+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa64004f4a0 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.290+0000 7fa648ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:09.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.290+0000 7fa648ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:09.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.615+0000 7fa652463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:09.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.615+0000 7fa652463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:09.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.615+0000 7fa652463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:09.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.615+0000 7fa652463640 1 -- 172.21.15.155:0/2531949081 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa62c005330 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.616+0000 7fa648ff9640 1 -- 172.21.15.155:0/2531949081 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fa64004f4a0 con 0x7fa64c0a9ac0 2024-03-29T11:37:09.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.616+0000 7fa648ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:09.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.616+0000 7fa648ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:09.618 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:37:09.618 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:09.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.622+0000 7fa652463640 10 librados: watch_flush enter 2024-03-29T11:37:09.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.622+0000 7fa652463640 10 librados: watch_flush exit 2024-03-29T11:37:09.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.622+0000 7fa652463640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-29T11:37:09.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.622+0000 7fa652463640 10 client.4326.objecter successfully canceled tick 2024-03-29T11:37:09.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.622+0000 7fa652463640 1 -- 172.21.15.155:0/2531949081 >> v1:172.21.15.155:6816/1760814708 conn(0x7fa63806e2a0 legacy=0x7fa638070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:09.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.623+0000 7fa652463640 10 monclient: shutdown 2024-03-29T11:37:09.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.623+0000 7fa652463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:09.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.623+0000 7fa652463640 1 -- 172.21.15.155:0/2531949081 >> v1:172.21.15.155:6789/0 conn(0x7fa64c0a9ac0 legacy=0x7fa64c070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:09.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.623+0000 7fa652463640 1 -- 172.21.15.155:0/2531949081 shutdown_connections 2024-03-29T11:37:09.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.623+0000 7fa652463640 1 -- 172.21.15.155:0/2531949081 >> 172.21.15.155:0/2531949081 conn(0x7fa64c0b35e0 msgr2=0x7fa64c13b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:09.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.623+0000 7fa652463640 1 -- 172.21.15.155:0/2531949081 shutdown_connections 2024-03-29T11:37:09.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.624+0000 7fa652463640 1 -- 172.21.15.155:0/2531949081 wait complete. 2024-03-29T11:37:09.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:09.626+0000 7fa652463640 1 librados: shutdown 2024-03-29T11:37:09.641 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:16.448 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:16.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.601+0000 7f1b6704a640 10 monclient: get_monmap_and_config 2024-03-29T11:37:16.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.601+0000 7f1b6704a640 10 monclient: build_initial_monmap 2024-03-29T11:37:16.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.601+0000 7f1b6704a640 10 monclient: monmap: 2024-03-29T11:37:16.600 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:16.600 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:16.600 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:37:16.601654+0000 2024-03-29T11:37:16.600 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:37:16.601654+0000 2024-03-29T11:37:16.600 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:16.601 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:16.601 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:16.601 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:37:16.601 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:37:16.601 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:16.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.604+0000 7f1b6704a640 1 Processor -- start 2024-03-29T11:37:16.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.604+0000 7f1b6704a640 1 -- start start 2024-03-29T11:37:16.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.604+0000 7f1b6704a640 10 monclient: init 2024-03-29T11:37:16.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.605+0000 7f1b6704a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:16.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.605+0000 7f1b6704a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:16.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.605+0000 7f1b6704a640 10 monclient(hunting): picked mon.noname-c con 0x7f1b600b1f60 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:16.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.605+0000 7f1b6704a640 10 monclient(hunting): picked mon.noname-b con 0x7f1b60071eb0 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:16.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.606+0000 7f1b6704a640 10 monclient(hunting): picked mon.noname-a con 0x7f1b60143d10 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:16.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.606+0000 7f1b6704a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6006ef80 con 0x7f1b60143d10 2024-03-29T11:37:16.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.606+0000 7f1b6704a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6006f260 con 0x7f1b600b1f60 2024-03-29T11:37:16.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.606+0000 7f1b6704a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b600b40c0 con 0x7f1b60071eb0 2024-03-29T11:37:16.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.606+0000 7f1b6704a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:16.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.606+0000 7f1b6704a640 10 monclient(hunting): authenticate will time out at 1018.732328s 2024-03-29T11:37:16.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.609+0000 7f1b64de6640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f1b600b1f60 0x7f1b60143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:57183/0 (socket says 172.21.15.155:57183) 2024-03-29T11:37:16.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.609+0000 7f1b64de6640 1 -- 172.21.15.155:0/3896519877 learned_addr learned my addr 172.21.15.155:0/3896519877 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:16.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 1 -- 172.21.15.155:0/3896519877 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998568205 0 0) 0x7f1b600b40c0 con 0x7f1b60071eb0 2024-03-29T11:37:16.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:16.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:16.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:16.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 1 -- 172.21.15.155:0/3896519877 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b48003520 con 0x7f1b60071eb0 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 1 -- 172.21.15.155:0/3896519877 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617823196 0 0) 0x7f1b6006f260 con 0x7f1b600b1f60 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.610+0000 7f1b5f7fe640 1 -- 172.21.15.155:0/3896519877 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b600b40c0 con 0x7f1b600b1f60 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 1 -- 172.21.15.155:0/3896519877 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3378466135 0 0) 0x7f1b48003520 con 0x7f1b60071eb0 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 1 -- 172.21.15.155:0/3896519877 >> v1:172.21.15.183:6789/0 conn(0x7f1b600b1f60 legacy=0x7f1b60143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 1 -- 172.21.15.155:0/3896519877 >> v1:172.21.15.155:6789/0 conn(0x7f1b60143d10 legacy=0x7f1b60146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:37:16.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 1 -- 172.21.15.155:0/3896519877 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b60155d70 con 0x7f1b60071eb0 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 1 -- 172.21.15.155:0/3896519877 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1b54001690 con 0x7f1b60071eb0 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 10 monclient: dump: 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:16.614 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 1 -- 172.21.15.155:0/3896519877 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b540017f0 con 0x7f1b60071eb0 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 1 -- 172.21.15.155:0/3896519877 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1b54001c00 con 0x7f1b60071eb0 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:16.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b5f7fe640 10 monclient: dump: 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b6704a640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:37:16.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b6704a640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:16.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.611+0000 7f1b6704a640 4 set_mon_vals no callback set 2024-03-29T11:37:16.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.612+0000 7f1b6704a640 10 monclient: shutdown 2024-03-29T11:37:16.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.612+0000 7f1b6704a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:16.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.612+0000 7f1b6704a640 1 -- 172.21.15.155:0/3896519877 >> v1:172.21.15.155:6790/0 conn(0x7f1b60071eb0 legacy=0x7f1b60146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.613+0000 7f1b6704a640 1 -- 172.21.15.155:0/3896519877 shutdown_connections 2024-03-29T11:37:16.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.613+0000 7f1b6704a640 1 -- 172.21.15.155:0/3896519877 >> 172.21.15.155:0/3896519877 conn(0x7f1b60066a00 msgr2=0x7f1b60066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:16.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.614+0000 7f1b6704a640 1 -- 172.21.15.155:0/3896519877 shutdown_connections 2024-03-29T11:37:16.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.615+0000 7f1b6704a640 1 -- 172.21.15.155:0/3896519877 wait complete. 2024-03-29T11:37:16.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.616+0000 7f1b6704a640 10 monclient: build_initial_monmap 2024-03-29T11:37:16.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.616+0000 7f1b6704a640 10 monclient: monmap: 2024-03-29T11:37:16.618 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:16.618 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:16.618 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:37:16.618 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.616+0000 7f1b6704a640 1 librados: starting msgr at 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.616+0000 7f1b6704a640 1 librados: starting objecter 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.617+0000 7f1b6704a640 1 Processor -- start 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.617+0000 7f1b6704a640 1 -- start start 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.617+0000 7f1b6704a640 1 librados: setting wanted keys 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.617+0000 7f1b6704a640 1 librados: calling monclient init 2024-03-29T11:37:16.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.617+0000 7f1b6704a640 10 monclient: init 2024-03-29T11:37:16.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.618+0000 7f1b6704a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:16.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.618+0000 7f1b6704a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:16.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.619+0000 7f1b6704a640 10 monclient(hunting): picked mon.noname-c con 0x7f1b600b1f60 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:16.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.621+0000 7f1b6704a640 10 monclient(hunting): picked mon.noname-a con 0x7f1b60143d10 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:16.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.622+0000 7f1b5ffff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f1b60143d10 0x7f1b601329a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:43851/0 (socket says 172.21.15.155:43851) 2024-03-29T11:37:16.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.622+0000 7f1b64de6640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f1b600b1f60 0x7f1b60139210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:60789/0 (socket says 172.21.15.155:60789) 2024-03-29T11:37:16.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.624+0000 7f1b5ffff640 1 -- 172.21.15.155:0/3840297113 learned_addr learned my addr 172.21.15.155:0/3840297113 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:16.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.625+0000 7f1b6704a640 10 monclient(hunting): picked mon.noname-b con 0x7f1b601545c0 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:16.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.625+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b601337c0 con 0x7f1b60143d10 2024-03-29T11:37:16.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.625+0000 7f1b5d7fa640 10 client.?.objecter ms_handle_connect 0x7f1b60143d10 2024-03-29T11:37:16.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.625+0000 7f1b5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:16.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.625+0000 7f1b5d7fa640 10 client.?.objecter ms_handle_connect 0x7f1b600b1f60 2024-03-29T11:37:16.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.625+0000 7f1b5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:16.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.625+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b600ad000 con 0x7f1b601545c0 2024-03-29T11:37:16.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.626+0000 7f1b5d7fa640 10 client.?.objecter ms_handle_connect 0x7f1b601545c0 2024-03-29T11:37:16.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.626+0000 7f1b5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:16.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.626+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261198620 0 0) 0x7f1b601337c0 con 0x7f1b60143d10 2024-03-29T11:37:16.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.625+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b600ab530 con 0x7f1b600b1f60 2024-03-29T11:37:16.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.626+0000 7f1b6704a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:16.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.626+0000 7f1b6704a640 10 monclient(hunting): authenticate will time out at 1018.752017s 2024-03-29T11:37:16.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:16.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:16.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:16.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:37:16.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b4c004eb0 con 0x7f1b60143d10 2024-03-29T11:37:16.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620640345 0 0) 0x7f1b600ad000 con 0x7f1b601545c0 2024-03-29T11:37:16.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:16.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:16.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:16.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:37:16.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b601337c0 con 0x7f1b601545c0 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.627+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200689465 0 0) 0x7f1b600ab530 con 0x7f1b600b1f60 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b600ad000 con 0x7f1b600b1f60 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3538982501 0 0) 0x7f1b4c004eb0 con 0x7f1b60143d10 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b600ab530 con 0x7f1b60143d10 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1b54001690 con 0x7f1b60143d10 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:16.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient(hunting): dump: 2024-03-29T11:37:16.629 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:16.629 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:16.629 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:16.629 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:16.629 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:16.629 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:16.629 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:16.629 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:16.629 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:16.629 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:16.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:16.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2143371697 0 0) 0x7f1b600ab530 con 0x7f1b60143d10 2024-03-29T11:37:16.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 >> v1:172.21.15.155:6790/0 conn(0x7f1b600b1f60 legacy=0x7f1b60139210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 >> v1:172.21.15.183:6789/0 conn(0x7f1b601545c0 legacy=0x7f1b601330b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:16.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 1 monclient: found mon.a 2024-03-29T11:37:16.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:16.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b60152840 con 0x7f1b60143d10 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b6704a640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b6704a640 10 monclient: _renew_subs 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b540039a0 con 0x7f1b60143d10 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b6704a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.628+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b60152a30 con 0x7f1b60143d10 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.629+0000 7f1b6704a640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.629+0000 7f1b5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.629+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1b54001cc0 con 0x7f1b60143d10 2024-03-29T11:37:16.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.629+0000 7f1b5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.629+0000 7f1b5d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.629+0000 7f1b5d7fa640 10 monclient: dump: 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:16.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.629+0000 7f1b6704a640 10 monclient: _renew_subs 2024-03-29T11:37:16.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.629+0000 7f1b6704a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:16.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.629+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b600abc30 con 0x7f1b60143d10 2024-03-29T11:37:16.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.629+0000 7f1b6704a640 1 librados: init done 2024-03-29T11:37:16.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.629+0000 7f1b5effd640 4 set_mon_vals no callback set 2024-03-29T11:37:16.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.631+0000 7f1b6704a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:16.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.631+0000 7f1b6704a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:16.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.631+0000 7f1b6704a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:16.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.631+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b60075f10 con 0x7f1b60143d10 2024-03-29T11:37:16.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.631+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f1b54003df0 con 0x7f1b60143d10 2024-03-29T11:37:16.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.632+0000 7f1b64de6640 10 monclient: get_auth_request con 0x7f1b4c06da20 auth_method 0 2024-03-29T11:37:16.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.633+0000 7f1b5d7fa640 10 client.4329.objecter ms_handle_connect 0x7f1b4c06da20 2024-03-29T11:37:16.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.633+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f1b540849d0 con 0x7f1b60143d10 2024-03-29T11:37:16.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.633+0000 7f1b5d7fa640 10 client.4329.objecter ms_dispatch 0x7f1b60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:16.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.633+0000 7f1b5d7fa640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:16.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.633+0000 7f1b5d7fa640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:16.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.633+0000 7f1b5d7fa640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-29T11:37:16.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.641+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b540502b0 con 0x7f1b60143d10 2024-03-29T11:37:16.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.641+0000 7f1b5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:16.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.641+0000 7f1b5d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:16.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.972+0000 7f1b6704a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:16.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.972+0000 7f1b6704a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:16.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.972+0000 7f1b6704a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:16.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.972+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b601529d0 con 0x7f1b60143d10 2024-03-29T11:37:16.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.975+0000 7f1b5d7fa640 1 -- 172.21.15.155:0/3840297113 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f1b540521e0 con 0x7f1b60143d10 2024-03-29T11:37:16.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.975+0000 7f1b5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:16.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.975+0000 7f1b5d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:16.975 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:37:16.975 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:16.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.979+0000 7f1b6704a640 10 librados: watch_flush enter 2024-03-29T11:37:16.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.980+0000 7f1b6704a640 10 librados: watch_flush exit 2024-03-29T11:37:16.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.980+0000 7f1b6704a640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-29T11:37:16.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.980+0000 7f1b6704a640 10 client.4329.objecter successfully canceled tick 2024-03-29T11:37:16.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.981+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 >> v1:172.21.15.155:6816/1760814708 conn(0x7f1b4c06da20 legacy=0x7f1b4c06fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.981+0000 7f1b6704a640 10 monclient: shutdown 2024-03-29T11:37:16.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.981+0000 7f1b6704a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:16.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.981+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 >> v1:172.21.15.155:6789/0 conn(0x7f1b60143d10 legacy=0x7f1b601329a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.981+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 shutdown_connections 2024-03-29T11:37:16.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.981+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 >> 172.21.15.155:0/3840297113 conn(0x7f1b60066a00 msgr2=0x7f1b60066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:16.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.982+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 shutdown_connections 2024-03-29T11:37:16.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.982+0000 7f1b6704a640 1 -- 172.21.15.155:0/3840297113 wait complete. 2024-03-29T11:37:16.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:16.982+0000 7f1b6704a640 1 librados: shutdown 2024-03-29T11:37:16.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:23.805 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:23.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.955+0000 7f47ba363640 10 monclient: get_monmap_and_config 2024-03-29T11:37:23.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.955+0000 7f47ba363640 10 monclient: build_initial_monmap 2024-03-29T11:37:23.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.955+0000 7f47ba363640 10 monclient: monmap: 2024-03-29T11:37:23.955 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:23.955 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:23.955 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:37:23.955886+0000 2024-03-29T11:37:23.955 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:37:23.955886+0000 2024-03-29T11:37:23.955 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:23.955 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:23.956 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:23.956 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:37:23.956 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:37:23.956 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:23.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.959+0000 7f47ba363640 1 Processor -- start 2024-03-29T11:37:23.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.960+0000 7f47ba363640 1 -- start start 2024-03-29T11:37:23.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.960+0000 7f47ba363640 10 monclient: init 2024-03-29T11:37:23.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.961+0000 7f47ba363640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:23.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.961+0000 7f47ba363640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:23.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.961+0000 7f47ba363640 10 monclient(hunting): picked mon.noname-a con 0x7f47b41422b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:23.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.961+0000 7f47ba363640 10 monclient(hunting): picked mon.noname-c con 0x7f47b4142690 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:23.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.961+0000 7f47ba363640 10 monclient(hunting): picked mon.noname-b con 0x7f47b40b17d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:23.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.961+0000 7f47ba363640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b4142c80 con 0x7f47b41422b0 2024-03-29T11:37:23.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.961+0000 7f47ba363640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b40aae40 con 0x7f47b4142690 2024-03-29T11:37:23.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.961+0000 7f47ba363640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b413fbd0 con 0x7f47b40b17d0 2024-03-29T11:37:23.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.961+0000 7f47ba363640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:23.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.961+0000 7f47ba363640 10 monclient(hunting): authenticate will time out at 1026.087245s 2024-03-29T11:37:23.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.964+0000 7f47b8900640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f47b40b17d0 0x7f47b40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:34603/0 (socket says 172.21.15.155:34603) 2024-03-29T11:37:23.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.964+0000 7f47b8900640 1 -- 172.21.15.155:0/1607410026 learned_addr learned my addr 172.21.15.155:0/1607410026 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:23.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.964+0000 7f47b37fe640 1 -- 172.21.15.155:0/1607410026 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336309873 0 0) 0x7f47b413fbd0 con 0x7f47b40b17d0 2024-03-29T11:37:23.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.965+0000 7f47b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:23.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.965+0000 7f47b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:23.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.965+0000 7f47b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:23.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.965+0000 7f47b37fe640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:37:23.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.965+0000 7f47b37fe640 1 -- 172.21.15.155:0/1607410026 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47a0003520 con 0x7f47b40b17d0 2024-03-29T11:37:23.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 1 -- 172.21.15.155:0/1607410026 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 356716389 0 0) 0x7f47a0003520 con 0x7f47b40b17d0 2024-03-29T11:37:23.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 1 -- 172.21.15.155:0/1607410026 >> v1:172.21.15.183:6789/0 conn(0x7f47b4142690 legacy=0x7f47b40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 1 -- 172.21.15.155:0/1607410026 >> v1:172.21.15.155:6789/0 conn(0x7f47b41422b0 legacy=0x7f47b413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:23.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 1 monclient: found mon.noname-b 2024-03-29T11:37:23.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:23.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:37:23.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 1 -- 172.21.15.155:0/1607410026 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b4151940 con 0x7f47b40b17d0 2024-03-29T11:37:23.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:23.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:23.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 1 -- 172.21.15.155:0/1607410026 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f47ac0019f0 con 0x7f47b40b17d0 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.966+0000 7f47b37fe640 10 monclient: dump: 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:23.972 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.969+0000 7f47ba363640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.969+0000 7f47ba363640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.969+0000 7f47b37fe640 1 -- 172.21.15.155:0/1607410026 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47ac001b50 con 0x7f47b40b17d0 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.969+0000 7f47b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.969+0000 7f47ba363640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.969+0000 7f47ba363640 4 set_mon_vals no callback set 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.969+0000 7f47b37fe640 1 -- 172.21.15.155:0/1607410026 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f47ac001f20 con 0x7f47b40b17d0 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.969+0000 7f47b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.969+0000 7f47b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.969+0000 7f47b37fe640 10 monclient: dump: 2024-03-29T11:37:23.973 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:23.974 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:23.974 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:23.974 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:23.974 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.974 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:23.974 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:23.974 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:23.974 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:23.978 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:23.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.970+0000 7f47ba363640 10 monclient: shutdown 2024-03-29T11:37:23.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.970+0000 7f47ba363640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:23.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.970+0000 7f47ba363640 1 -- 172.21.15.155:0/1607410026 >> v1:172.21.15.155:6790/0 conn(0x7f47b40b17d0 legacy=0x7f47b40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.970+0000 7f47ba363640 1 -- 172.21.15.155:0/1607410026 shutdown_connections 2024-03-29T11:37:23.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.970+0000 7f47ba363640 1 -- 172.21.15.155:0/1607410026 >> 172.21.15.155:0/1607410026 conn(0x7f47b4066a00 msgr2=0x7f47b4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:23.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.971+0000 7f47ba363640 1 -- 172.21.15.155:0/1607410026 shutdown_connections 2024-03-29T11:37:23.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.971+0000 7f47ba363640 1 -- 172.21.15.155:0/1607410026 wait complete. 2024-03-29T11:37:23.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.972+0000 7f47ba363640 10 monclient: build_initial_monmap 2024-03-29T11:37:23.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.972+0000 7f47ba363640 10 monclient: monmap: 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.973+0000 7f47ba363640 1 librados: starting msgr at 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.973+0000 7f47ba363640 1 librados: starting objecter 2024-03-29T11:37:23.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.973+0000 7f47ba363640 1 Processor -- start 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.973+0000 7f47ba363640 1 -- start start 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.973+0000 7f47ba363640 1 librados: setting wanted keys 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.973+0000 7f47ba363640 1 librados: calling monclient init 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.973+0000 7f47ba363640 10 monclient: init 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.974+0000 7f47ba363640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.974+0000 7f47ba363640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.974+0000 7f47ba363640 10 monclient(hunting): picked mon.noname-a con 0x7f47b40b17d0 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.974+0000 7f47ba363640 10 monclient(hunting): picked mon.noname-b con 0x7f47b41422b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.974+0000 7f47ba363640 10 monclient(hunting): picked mon.noname-c con 0x7f47b4142690 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.974+0000 7f47ba363640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b413c650 con 0x7f47b40b17d0 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.974+0000 7f47ba363640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b40ac7a0 con 0x7f47b41422b0 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.974+0000 7f47ba363640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b40aca80 con 0x7f47b4142690 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.974+0000 7f47ba363640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:23.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.974+0000 7f47ba363640 10 monclient(hunting): authenticate will time out at 1026.100436s 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.977+0000 7f47b3fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f47b40b17d0 0x7f47b41e4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:45257/0 (socket says 172.21.15.155:45257) 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.977+0000 7f47b3fff640 1 -- 172.21.15.155:0/3200221953 learned_addr learned my addr 172.21.15.155:0/3200221953 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.978+0000 7f47b17fa640 10 client.?.objecter ms_handle_connect 0x7f47b40b17d0 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.978+0000 7f47b17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.978+0000 7f47b17fa640 10 client.?.objecter ms_handle_connect 0x7f47b41422b0 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.978+0000 7f47b17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.979+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975942839 0 0) 0x7f47b413c650 con 0x7f47b40b17d0 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.979+0000 7f47b17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.979+0000 7f47b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.979+0000 7f47b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.979+0000 7f47b17fa640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.979+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4794004eb0 con 0x7f47b40b17d0 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.980+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039882455 0 0) 0x7f47b40ac7a0 con 0x7f47b41422b0 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.980+0000 7f47b17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:23.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.980+0000 7f47b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:23.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.980+0000 7f47b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:23.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.980+0000 7f47b17fa640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:37:23.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.980+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47940051d0 con 0x7f47b41422b0 2024-03-29T11:37:23.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 client.?.objecter ms_handle_connect 0x7f47b4142690 2024-03-29T11:37:23.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:23.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 888829893 0 0) 0x7f4794004eb0 con 0x7f47b40b17d0 2024-03-29T11:37:23.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47b40ac7a0 con 0x7f47b40b17d0 2024-03-29T11:37:23.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154628044 0 0) 0x7f47940051d0 con 0x7f47b41422b0 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4794004eb0 con 0x7f47b41422b0 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390472779 0 0) 0x7f47b40aca80 con 0x7f47b4142690 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47940051d0 con 0x7f47b4142690 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f479c0019f0 con 0x7f47b40b17d0 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient(hunting): dump: 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:23.988 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f47a4001810 con 0x7f47b41422b0 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:23.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient(hunting): dump: 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1122085852 0 0) 0x7f4794004eb0 con 0x7f47b41422b0 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 >> v1:172.21.15.155:6790/0 conn(0x7f47b4142690 legacy=0x7f47b413bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 >> v1:172.21.15.155:6789/0 conn(0x7f47b40b17d0 legacy=0x7f47b41e4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:23.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 monclient: found mon.b 2024-03-29T11:37:23.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:23.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:37:23.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b41e43f0 con 0x7f47b41422b0 2024-03-29T11:37:23.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:23.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:23.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:23.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47ba363640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:37:23.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47ba363640 10 monclient: _renew_subs 2024-03-29T11:37:23.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47ba363640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:37:23.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47ba363640 1 -- 172.21.15.155:0/3200221953 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b41e5020 con 0x7f47b41422b0 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.981+0000 7f47ba363640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.982+0000 7f47ba363640 10 monclient: _renew_subs 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.982+0000 7f47ba363640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.982+0000 7f47ba363640 1 -- 172.21.15.155:0/3200221953 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b406f0c0 con 0x7f47b41422b0 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.982+0000 7f47ba363640 1 librados: init done 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.982+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47a4001ca0 con 0x7f47b41422b0 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.982+0000 7f47b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.982+0000 7f47b27fc640 4 set_mon_vals no callback set 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.983+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f47a40037b0 con 0x7f47b41422b0 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.983+0000 7f47b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.983+0000 7f47b17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.983+0000 7f47b17fa640 10 monclient: dump: 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:23.992 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.984+0000 7f47ba363640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.984+0000 7f47ba363640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.984+0000 7f47ba363640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.984+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f47a4003990 con 0x7f47b41422b0 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.984+0000 7f47ba363640 1 -- 172.21.15.155:0/3200221953 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4798005080 con 0x7f47b41422b0 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.986+0000 7f47b3fff640 10 monclient: get_auth_request con 0x7f479406e3d0 auth_method 0 2024-03-29T11:37:23.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.986+0000 7f47b17fa640 10 client.4330.objecter ms_handle_connect 0x7f479406e3d0 2024-03-29T11:37:23.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.986+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f47a40035f0 con 0x7f47b41422b0 2024-03-29T11:37:23.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.987+0000 7f47b17fa640 10 client.4330.objecter ms_dispatch 0x7f47b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:23.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.987+0000 7f47b17fa640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:23.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.987+0000 7f47b17fa640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:23.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.987+0000 7f47b17fa640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T11:37:23.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.997+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47a404f5a0 con 0x7f47b41422b0 2024-03-29T11:37:23.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.997+0000 7f47b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:23.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:23.997+0000 7f47b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:24.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.325+0000 7f47ba363640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:24.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.325+0000 7f47ba363640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:24.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.325+0000 7f47ba363640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:37:24.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.325+0000 7f47ba363640 1 -- 172.21.15.155:0/3200221953 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4798005330 con 0x7f47b41422b0 2024-03-29T11:37:24.325 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:37:24.325 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:24.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.326+0000 7f47b17fa640 1 -- 172.21.15.155:0/3200221953 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f47a404f740 con 0x7f47b41422b0 2024-03-29T11:37:24.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.326+0000 7f47b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:24.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.326+0000 7f47b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:24.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.329+0000 7f47ba363640 10 librados: watch_flush enter 2024-03-29T11:37:24.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.332+0000 7f47ba363640 10 librados: watch_flush exit 2024-03-29T11:37:24.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.332+0000 7f47ba363640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T11:37:24.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.332+0000 7f47ba363640 10 client.4330.objecter successfully canceled tick 2024-03-29T11:37:24.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.332+0000 7f47ba363640 1 -- 172.21.15.155:0/3200221953 >> v1:172.21.15.155:6816/1760814708 conn(0x7f479406e3d0 legacy=0x7f4794070890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:24.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.332+0000 7f47ba363640 10 monclient: shutdown 2024-03-29T11:37:24.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.332+0000 7f47ba363640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:24.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.332+0000 7f47ba363640 1 -- 172.21.15.155:0/3200221953 >> v1:172.21.15.183:6789/0 conn(0x7f47b41422b0 legacy=0x7f47b413b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:24.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.333+0000 7f47ba363640 1 -- 172.21.15.155:0/3200221953 shutdown_connections 2024-03-29T11:37:24.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.333+0000 7f47ba363640 1 -- 172.21.15.155:0/3200221953 >> 172.21.15.155:0/3200221953 conn(0x7f47b4066a00 msgr2=0x7f47b41413f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:24.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.333+0000 7f47ba363640 1 -- 172.21.15.155:0/3200221953 shutdown_connections 2024-03-29T11:37:24.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.333+0000 7f47ba363640 1 -- 172.21.15.155:0/3200221953 wait complete. 2024-03-29T11:37:24.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:24.333+0000 7f47ba363640 1 librados: shutdown 2024-03-29T11:37:24.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:31.161 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:31.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.320+0000 7f2646e8f640 10 monclient: get_monmap_and_config 2024-03-29T11:37:31.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.320+0000 7f2646e8f640 10 monclient: build_initial_monmap 2024-03-29T11:37:31.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.320+0000 7f2646e8f640 10 monclient: monmap: 2024-03-29T11:37:31.321 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:31.321 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:31.321 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:37:31.321817+0000 2024-03-29T11:37:31.321 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:37:31.321817+0000 2024-03-29T11:37:31.321 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:31.322 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:31.322 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:31.322 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:37:31.322 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:37:31.322 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:31.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.323+0000 7f2646e8f640 1 Processor -- start 2024-03-29T11:37:31.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.323+0000 7f2646e8f640 1 -- start start 2024-03-29T11:37:31.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.323+0000 7f2646e8f640 10 monclient: init 2024-03-29T11:37:31.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.324+0000 7f2646e8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:31.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.324+0000 7f2646e8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:31.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.324+0000 7f2646e8f640 10 monclient(hunting): picked mon.noname-a con 0x7f26400b0770 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:31.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.324+0000 7f2646e8f640 10 monclient(hunting): picked mon.noname-b con 0x7f264014a700 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:31.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f2646e8f640 10 monclient(hunting): picked mon.noname-c con 0x7f26400ac5a0 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:31.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f2646e8f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26400aba70 con 0x7f26400b0770 2024-03-29T11:37:31.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f2646e8f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26400aecb0 con 0x7f26400ac5a0 2024-03-29T11:37:31.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f2646e8f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26400af150 con 0x7f264014a700 2024-03-29T11:37:31.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f2646e8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:31.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f2646e8f640 10 monclient(hunting): authenticate will time out at 1033.455313s 2024-03-29T11:37:31.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f264542c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f26400ac5a0 0x7f2640151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:48431/0 (socket says 172.21.15.155:48431) 2024-03-29T11:37:31.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f264542c640 1 -- 172.21.15.155:0/208482895 learned_addr learned my addr 172.21.15.155:0/208482895 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:31.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f263f7fe640 1 -- 172.21.15.155:0/208482895 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264815218 0 0) 0x7f26400aecb0 con 0x7f26400ac5a0 2024-03-29T11:37:31.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f263f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:31.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f263f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:31.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f263f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:31.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f263f7fe640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:37:31.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.328+0000 7f263f7fe640 1 -- 172.21.15.155:0/208482895 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f262c003520 con 0x7f26400ac5a0 2024-03-29T11:37:31.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.329+0000 7f263f7fe640 1 -- 172.21.15.155:0/208482895 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2523188557 0 0) 0x7f262c003520 con 0x7f26400ac5a0 2024-03-29T11:37:31.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.329+0000 7f263f7fe640 1 -- 172.21.15.155:0/208482895 >> v1:172.21.15.155:6790/0 conn(0x7f264014a700 legacy=0x7f26400abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:31.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 1 -- 172.21.15.155:0/208482895 >> v1:172.21.15.155:6789/0 conn(0x7f26400b0770 legacy=0x7f2640149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:31.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:37:31.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:31.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:37:31.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 1 -- 172.21.15.155:0/208482895 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26400ac9d0 con 0x7f26400ac5a0 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 1 -- 172.21.15.155:0/208482895 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2638001a30 con 0x7f26400ac5a0 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.332+0000 7f263f7fe640 10 monclient: dump: 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.335 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:31.336 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:31.336 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:31.336 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:31.336 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:31.336 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:31.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.333+0000 7f263f7fe640 1 -- 172.21.15.155:0/208482895 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2638001c00 con 0x7f26400ac5a0 2024-03-29T11:37:31.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.333+0000 7f263f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:31.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.333+0000 7f263f7fe640 1 -- 172.21.15.155:0/208482895 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2638002010 con 0x7f26400ac5a0 2024-03-29T11:37:31.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.333+0000 7f263f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.333+0000 7f263f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.333+0000 7f263f7fe640 10 monclient: dump: 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.333+0000 7f2646e8f640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:37:31.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.333+0000 7f2646e8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:31.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.333+0000 7f2646e8f640 4 set_mon_vals no callback set 2024-03-29T11:37:31.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.334+0000 7f2646e8f640 10 monclient: shutdown 2024-03-29T11:37:31.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.334+0000 7f2646e8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:31.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.334+0000 7f2646e8f640 1 -- 172.21.15.155:0/208482895 >> v1:172.21.15.183:6789/0 conn(0x7f26400ac5a0 legacy=0x7f2640151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.334+0000 7f2646e8f640 1 -- 172.21.15.155:0/208482895 shutdown_connections 2024-03-29T11:37:31.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.334+0000 7f2646e8f640 1 -- 172.21.15.155:0/208482895 >> 172.21.15.155:0/208482895 conn(0x7f264006b660 msgr2=0x7f264006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:31.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.335+0000 7f2646e8f640 1 -- 172.21.15.155:0/208482895 shutdown_connections 2024-03-29T11:37:31.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.335+0000 7f2646e8f640 1 -- 172.21.15.155:0/208482895 wait complete. 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.335+0000 7f2646e8f640 10 monclient: build_initial_monmap 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.336+0000 7f2646e8f640 10 monclient: monmap: 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:37:31.341 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:31.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.336+0000 7f2646e8f640 1 librados: starting msgr at 2024-03-29T11:37:31.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.336+0000 7f2646e8f640 1 librados: starting objecter 2024-03-29T11:37:31.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.336+0000 7f2646e8f640 1 Processor -- start 2024-03-29T11:37:31.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.336+0000 7f2646e8f640 1 -- start start 2024-03-29T11:37:31.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.337+0000 7f2646e8f640 1 librados: setting wanted keys 2024-03-29T11:37:31.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.337+0000 7f2646e8f640 1 librados: calling monclient init 2024-03-29T11:37:31.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.337+0000 7f2646e8f640 10 monclient: init 2024-03-29T11:37:31.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.338+0000 7f2646e8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:31.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.338+0000 7f2646e8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:31.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.338+0000 7f2646e8f640 10 monclient(hunting): picked mon.noname-a con 0x7f26400ac5a0 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:31.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.341+0000 7f263ffff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f26400b0770 0x7f2640144f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:37815/0 (socket says 172.21.15.155:37815) 2024-03-29T11:37:31.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.341+0000 7f2646e8f640 10 monclient(hunting): picked mon.noname-c con 0x7f26400b0770 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:31.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.341+0000 7f2646e8f640 10 monclient(hunting): picked mon.noname-b con 0x7f264014a700 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:31.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.341+0000 7f2646e8f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26401283e0 con 0x7f26400ac5a0 2024-03-29T11:37:31.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.341+0000 7f2646e8f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26401262d0 con 0x7f264014a700 2024-03-29T11:37:31.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.341+0000 7f2646e8f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2640125f80 con 0x7f26400b0770 2024-03-29T11:37:31.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.341+0000 7f2646e8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:31.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.341+0000 7f2646e8f640 10 monclient(hunting): authenticate will time out at 1033.468757s 2024-03-29T11:37:31.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.341+0000 7f263ffff640 1 -- 172.21.15.155:0/1057796891 learned_addr learned my addr 172.21.15.155:0/1057796891 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:31.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 10 client.?.objecter ms_handle_connect 0x7f26400b0770 2024-03-29T11:37:31.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:31.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 10 client.?.objecter ms_handle_connect 0x7f26400ac5a0 2024-03-29T11:37:31.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:31.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 10 client.?.objecter ms_handle_connect 0x7f264014a700 2024-03-29T11:37:31.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:31.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906155004 0 0) 0x7f26401262d0 con 0x7f264014a700 2024-03-29T11:37:31.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:31.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:31.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:31.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.345+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2620004eb0 con 0x7f264014a700 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733433269 0 0) 0x7f26401283e0 con 0x7f26400ac5a0 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26401262d0 con 0x7f26400ac5a0 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172356073 0 0) 0x7f2640125f80 con 0x7f26400b0770 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26401283e0 con 0x7f26400b0770 2024-03-29T11:37:31.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2089639320 0 0) 0x7f26401262d0 con 0x7f26400ac5a0 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2640125f80 con 0x7f26400ac5a0 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2628001870 con 0x7f26400ac5a0 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): dump: 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:31.354 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2311612087 0 0) 0x7f2620004eb0 con 0x7f264014a700 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26401262d0 con 0x7f264014a700 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2638001f10 con 0x7f264014a700 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): dump: 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:31.355 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 113289001 0 0) 0x7f26401283e0 con 0x7f26400b0770 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2620004eb0 con 0x7f26400b0770 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2630001690 con 0x7f26400b0770 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient(hunting): dump: 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:31.356 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2169708118 0 0) 0x7f2640125f80 con 0x7f26400ac5a0 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 >> v1:172.21.15.155:6790/0 conn(0x7f26400b0770 legacy=0x7f2640144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 >> v1:172.21.15.183:6789/0 conn(0x7f264014a700 legacy=0x7f26401e7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 monclient: found mon.a 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:31.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26401239d0 con 0x7f26400ac5a0 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.346+0000 7f263d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.347+0000 7f2646e8f640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.347+0000 7f2646e8f640 10 monclient: _renew_subs 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.347+0000 7f2646e8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.347+0000 7f2646e8f640 1 -- 172.21.15.155:0/1057796891 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26401273e0 con 0x7f26400ac5a0 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.347+0000 7f2646e8f640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.348+0000 7f2646e8f640 10 monclient: _renew_subs 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.348+0000 7f2646e8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.348+0000 7f2646e8f640 1 -- 172.21.15.155:0/1057796891 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26401dfc10 con 0x7f26400ac5a0 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.348+0000 7f2646e8f640 1 librados: init done 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.351+0000 7f2646e8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:31.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.351+0000 7f2646e8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:31.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.351+0000 7f2646e8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:31.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.351+0000 7f2646e8f640 1 -- 172.21.15.155:0/1057796891 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2640075f10 con 0x7f26400ac5a0 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.352+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2628001c80 con 0x7f26400ac5a0 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.352+0000 7f263d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.352+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f26280037b0 con 0x7f26400ac5a0 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.352+0000 7f263d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.352+0000 7f263d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.352+0000 7f263d7fa640 10 monclient: dump: 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:31.360 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.353+0000 7f263effd640 4 set_mon_vals no callback set 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.355+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f2628019780 con 0x7f26400ac5a0 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.356+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f2628083480 con 0x7f26400ac5a0 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.356+0000 7f263d7fa640 10 client.4341.objecter ms_dispatch 0x7f2640000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.356+0000 7f263d7fa640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.356+0000 7f263d7fa640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.356+0000 7f263d7fa640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.356+0000 7f263ffff640 10 monclient: get_auth_request con 0x7f262006e260 auth_method 0 2024-03-29T11:37:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.356+0000 7f263d7fa640 10 client.4341.objecter ms_handle_connect 0x7f262006e260 2024-03-29T11:37:31.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.362+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f262804edc0 con 0x7f26400ac5a0 2024-03-29T11:37:31.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.362+0000 7f263d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:31.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.362+0000 7f263d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:31.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.682+0000 7f2646e8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:31.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.682+0000 7f2646e8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:31.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.682+0000 7f2646e8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:31.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.682+0000 7f2646e8f640 1 -- 172.21.15.155:0/1057796891 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26400b2440 con 0x7f26400ac5a0 2024-03-29T11:37:31.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.684+0000 7f263d7fa640 1 -- 172.21.15.155:0/1057796891 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f2628003610 con 0x7f26400ac5a0 2024-03-29T11:37:31.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.684+0000 7f263d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:31.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.684+0000 7f263d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:31.685 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:37:31.685 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.688+0000 7f2646e8f640 10 librados: watch_flush enter 2024-03-29T11:37:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.688+0000 7f2646e8f640 10 librados: watch_flush exit 2024-03-29T11:37:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.688+0000 7f2646e8f640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-29T11:37:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.688+0000 7f2646e8f640 10 client.4341.objecter successfully canceled tick 2024-03-29T11:37:31.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.688+0000 7f2646e8f640 1 -- 172.21.15.155:0/1057796891 >> v1:172.21.15.155:6816/1760814708 conn(0x7f262006e260 legacy=0x7f2620070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.688+0000 7f2646e8f640 10 monclient: shutdown 2024-03-29T11:37:31.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.688+0000 7f2646e8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:31.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.688+0000 7f2646e8f640 1 -- 172.21.15.155:0/1057796891 >> v1:172.21.15.155:6789/0 conn(0x7f26400ac5a0 legacy=0x7f26401267f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.690+0000 7f2646e8f640 1 -- 172.21.15.155:0/1057796891 shutdown_connections 2024-03-29T11:37:31.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.690+0000 7f2646e8f640 1 -- 172.21.15.155:0/1057796891 >> 172.21.15.155:0/1057796891 conn(0x7f264006b660 msgr2=0x7f26400ae2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:31.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.691+0000 7f2646e8f640 1 -- 172.21.15.155:0/1057796891 shutdown_connections 2024-03-29T11:37:31.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.691+0000 7f2646e8f640 1 -- 172.21.15.155:0/1057796891 wait complete. 2024-03-29T11:37:31.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:31.691+0000 7f2646e8f640 1 librados: shutdown 2024-03-29T11:37:31.709 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:38.516 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:38.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.673+0000 7f39a3fff640 10 monclient: get_monmap_and_config 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.673+0000 7f39a3fff640 10 monclient: build_initial_monmap 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.673+0000 7f39a3fff640 10 monclient: monmap: 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:37:38.674779+0000 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:37:38.674779+0000 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:37:38.674 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:38.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.675+0000 7f39a3fff640 1 Processor -- start 2024-03-29T11:37:38.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.675+0000 7f39a3fff640 1 -- start start 2024-03-29T11:37:38.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.675+0000 7f39a3fff640 10 monclient: init 2024-03-29T11:37:38.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.676+0000 7f39a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:38.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.676+0000 7f39a3fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:38.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.676+0000 7f39a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f399c006e30 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:38.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.679+0000 7f39a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f399c035e00 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:38.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.679+0000 7f39a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f399c0c9670 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:38.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.682+0000 7f39a3fff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c035970 con 0x7f399c006e30 2024-03-29T11:37:38.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.682+0000 7f39a3fff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c036280 con 0x7f399c035e00 2024-03-29T11:37:38.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.682+0000 7f39a3fff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c035640 con 0x7f399c0c9670 2024-03-29T11:37:38.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.682+0000 7f39a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:38.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.682+0000 7f39a3fff640 10 monclient(hunting): authenticate will time out at 1040.809633s 2024-03-29T11:37:38.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.682+0000 7f39a37fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f399c0c9670 0x7f399c0d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:46829/0 (socket says 172.21.15.155:46829) 2024-03-29T11:37:38.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.682+0000 7f39a37fe640 1 -- 172.21.15.155:0/3579760415 learned_addr learned my addr 172.21.15.155:0/3579760415 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:38.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.683+0000 7f39a1ffb640 1 -- 172.21.15.155:0/3579760415 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193176286 0 0) 0x7f399c035640 con 0x7f399c0c9670 2024-03-29T11:37:38.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.683+0000 7f39a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:38.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.683+0000 7f39a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:38.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.683+0000 7f39a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:38.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.683+0000 7f39a1ffb640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:37:38.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.683+0000 7f39a1ffb640 1 -- 172.21.15.155:0/3579760415 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f398c003520 con 0x7f399c0c9670 2024-03-29T11:37:38.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 1 -- 172.21.15.155:0/3579760415 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2272561293 0 0) 0x7f398c003520 con 0x7f399c0c9670 2024-03-29T11:37:38.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 1 -- 172.21.15.155:0/3579760415 >> v1:172.21.15.183:6789/0 conn(0x7f399c035e00 legacy=0x7f399c0c8f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:37:38.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 1 -- 172.21.15.155:0/3579760415 >> v1:172.21.15.155:6789/0 conn(0x7f399c006e30 legacy=0x7f399c0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:38.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 1 -- 172.21.15.155:0/3579760415 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c034e20 con 0x7f399c0c9670 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a1ffb640 1 -- 172.21.15.155:0/3579760415 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3994001690 con 0x7f399c0c9670 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a3fff640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.684+0000 7f39a3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.685+0000 7f39a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.685+0000 7f39a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.685+0000 7f39a1ffb640 10 monclient: dump: 2024-03-29T11:37:38.687 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:38.688 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:38.688 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:38.688 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:38.688 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:38.688 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:38.688 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:38.688 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:38.688 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:38.688 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:38.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.685+0000 7f39a3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:38.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.689+0000 7f39a1ffb640 1 -- 172.21.15.155:0/3579760415 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3994001860 con 0x7f399c0c9670 2024-03-29T11:37:38.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.689+0000 7f39a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:38.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.689+0000 7f39a1ffb640 1 -- 172.21.15.155:0/3579760415 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3994001cb0 con 0x7f399c0c9670 2024-03-29T11:37:38.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.689+0000 7f39a3fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:38.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.689+0000 7f39a3fff640 4 set_mon_vals no callback set 2024-03-29T11:37:38.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.690+0000 7f39a3fff640 10 monclient: shutdown 2024-03-29T11:37:38.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.690+0000 7f39a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:38.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.690+0000 7f39a3fff640 1 -- 172.21.15.155:0/3579760415 >> v1:172.21.15.155:6790/0 conn(0x7f399c0c9670 legacy=0x7f399c0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:38.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.690+0000 7f39a1ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:37:38.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.690+0000 7f39a3fff640 1 -- 172.21.15.155:0/3579760415 shutdown_connections 2024-03-29T11:37:38.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.690+0000 7f39a3fff640 1 -- 172.21.15.155:0/3579760415 >> 172.21.15.155:0/3579760415 conn(0x7f399c005c00 msgr2=0x7f399c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:38.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.691+0000 7f39a3fff640 1 -- 172.21.15.155:0/3579760415 shutdown_connections 2024-03-29T11:37:38.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.691+0000 7f39a3fff640 1 -- 172.21.15.155:0/3579760415 wait complete. 2024-03-29T11:37:38.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.692+0000 7f39a3fff640 10 monclient: build_initial_monmap 2024-03-29T11:37:38.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.693+0000 7f39a3fff640 10 monclient: monmap: 2024-03-29T11:37:38.693 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:38.693 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:38.693 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:37:38.693 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:37:38.693 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:38.693 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:38.693 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:38.693 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:37:38.693 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:37:38.694 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:38.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.693+0000 7f39a3fff640 1 librados: starting msgr at 2024-03-29T11:37:38.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.696+0000 7f39a3fff640 1 librados: starting objecter 2024-03-29T11:37:38.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.697+0000 7f39a3fff640 1 Processor -- start 2024-03-29T11:37:38.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.697+0000 7f39a3fff640 1 -- start start 2024-03-29T11:37:38.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.697+0000 7f39a3fff640 1 librados: setting wanted keys 2024-03-29T11:37:38.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.697+0000 7f39a3fff640 1 librados: calling monclient init 2024-03-29T11:37:38.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.697+0000 7f39a3fff640 10 monclient: init 2024-03-29T11:37:38.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.698+0000 7f39a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:38.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.698+0000 7f39a3fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:38.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.701+0000 7f39a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f399c006e30 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:38.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.705+0000 7f39a27fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f399c035e00 0x7f399c033770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:35661/0 (socket says 172.21.15.155:35661) 2024-03-29T11:37:38.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.705+0000 7f39a2ffd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f399c006e30 0x7f399c0ceb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48239/0 (socket says 172.21.15.155:48239) 2024-03-29T11:37:38.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.705+0000 7f39a27fc640 1 -- 172.21.15.155:0/3725196020 learned_addr learned my addr 172.21.15.155:0/3725196020 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:38.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.705+0000 7f39a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f399c035e00 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:38.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.705+0000 7f39a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f399c0c9670 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:38.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.706+0000 7f398b7fe640 10 client.?.objecter ms_handle_connect 0x7f399c035e00 2024-03-29T11:37:38.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.706+0000 7f398b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:38.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.706+0000 7f398b7fe640 10 client.?.objecter ms_handle_connect 0x7f399c006e30 2024-03-29T11:37:38.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.706+0000 7f398b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:38.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.707+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c0cf380 con 0x7f399c006e30 2024-03-29T11:37:38.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.707+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c02d4d0 con 0x7f399c035e00 2024-03-29T11:37:38.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.707+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c02d7b0 con 0x7f399c0c9670 2024-03-29T11:37:38.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.707+0000 7f39a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:38.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.707+0000 7f39a3fff640 10 monclient(hunting): authenticate will time out at 1040.834395s 2024-03-29T11:37:38.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.707+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165421381 0 0) 0x7f399c0cf380 con 0x7f399c006e30 2024-03-29T11:37:38.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.707+0000 7f398b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:38.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.707+0000 7f398b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:38.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.707+0000 7f398b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:38.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.707+0000 7f398b7fe640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:37:38.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.708+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3990004eb0 con 0x7f399c006e30 2024-03-29T11:37:38.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.708+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443449336 0 0) 0x7f399c02d4d0 con 0x7f399c035e00 2024-03-29T11:37:38.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.708+0000 7f398b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:38.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.708+0000 7f398b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:38.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.708+0000 7f398b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:38.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.708+0000 7f398b7fe640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:37:38.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.708+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f399c0cf380 con 0x7f399c035e00 2024-03-29T11:37:38.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.709+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2512110888 0 0) 0x7f3990004eb0 con 0x7f399c006e30 2024-03-29T11:37:38.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.709+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f399c02d4d0 con 0x7f399c006e30 2024-03-29T11:37:38.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.709+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f39980018d0 con 0x7f399c006e30 2024-03-29T11:37:38.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.709+0000 7f398b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.709+0000 7f398b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.709+0000 7f398b7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.709+0000 7f398b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.709+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2530658204 0 0) 0x7f399c02d4d0 con 0x7f399c006e30 2024-03-29T11:37:38.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.709+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 >> v1:172.21.15.155:6790/0 conn(0x7f399c0c9670 legacy=0x7f399c16bad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:38.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.710+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 >> v1:172.21.15.183:6789/0 conn(0x7f399c035e00 legacy=0x7f399c033770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:38.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:38.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 1 monclient: found mon.a 2024-03-29T11:37:38.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:38.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:38.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c0ce0d0 con 0x7f399c006e30 2024-03-29T11:37:38.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:38.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:38.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:38.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f39a3fff640 5 monclient: authenticate success, global_id 4347 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f39a3fff640 10 monclient: _renew_subs 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f39a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399c0cfe90 con 0x7f399c006e30 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f39a3fff640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f39a3fff640 10 monclient: _renew_subs 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f39a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f399c02ddb0 con 0x7f399c006e30 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f39a3fff640 1 librados: init done 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3998001aa0 con 0x7f399c006e30 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3998003a60 con 0x7f399c006e30 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:38.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f398b7fe640 10 monclient: dump: 2024-03-29T11:37:38.717 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:38.717 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:38.717 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:38.717 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:38.717 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:38.717 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:38.717 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:38.717 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:38.717 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:38.717 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:38.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.714+0000 7f39a0ff9640 4 set_mon_vals no callback set 2024-03-29T11:37:38.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.715+0000 7f39a3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:38.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.715+0000 7f39a3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:38.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.715+0000 7f39a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:38.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.715+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3984005080 con 0x7f399c006e30 2024-03-29T11:37:38.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.716+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f3998003bc0 con 0x7f399c006e30 2024-03-29T11:37:38.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.717+0000 7f39a27fc640 10 monclient: get_auth_request con 0x7f399006da80 auth_method 0 2024-03-29T11:37:38.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.717+0000 7f398b7fe640 10 client.4347.objecter ms_handle_connect 0x7f399006da80 2024-03-29T11:37:38.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.717+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f3998084ab0 con 0x7f399c006e30 2024-03-29T11:37:38.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.717+0000 7f398b7fe640 10 client.4347.objecter ms_dispatch 0x7f39a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:38.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.717+0000 7f398b7fe640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:38.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.717+0000 7f398b7fe640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:38.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.717+0000 7f398b7fe640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T11:37:38.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.730+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39980501c0 con 0x7f399c006e30 2024-03-29T11:37:38.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.730+0000 7f398b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:38.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:38.730+0000 7f398b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:39.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.055+0000 7f39a3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:39.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.055+0000 7f39a3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:39.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.055+0000 7f39a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:37:39.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.055+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3984005330 con 0x7f399c006e30 2024-03-29T11:37:39.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.056+0000 7f398b7fe640 1 -- 172.21.15.155:0/3725196020 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f39980522c0 con 0x7f399c006e30 2024-03-29T11:37:39.056 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:37:39.056 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:39.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.056+0000 7f398b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:39.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.056+0000 7f398b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:39.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.059+0000 7f39a3fff640 10 librados: watch_flush enter 2024-03-29T11:37:39.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.059+0000 7f39a3fff640 10 librados: watch_flush exit 2024-03-29T11:37:39.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.059+0000 7f39a3fff640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T11:37:39.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.059+0000 7f39a3fff640 10 client.4347.objecter successfully canceled tick 2024-03-29T11:37:39.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.059+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 >> v1:172.21.15.155:6816/1760814708 conn(0x7f399006da80 legacy=0x7f399006ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:39.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.059+0000 7f39a3fff640 10 monclient: shutdown 2024-03-29T11:37:39.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.059+0000 7f39a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:39.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.059+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 >> v1:172.21.15.155:6789/0 conn(0x7f399c006e30 legacy=0x7f399c0ceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:39.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.060+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 shutdown_connections 2024-03-29T11:37:39.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.060+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 >> 172.21.15.155:0/3725196020 conn(0x7f399c005c00 msgr2=0x7f399c0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:39.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.061+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 shutdown_connections 2024-03-29T11:37:39.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.063+0000 7f39a3fff640 1 -- 172.21.15.155:0/3725196020 wait complete. 2024-03-29T11:37:39.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:39.067+0000 7f39a3fff640 1 librados: shutdown 2024-03-29T11:37:39.085 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:45.891 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:46.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.029+0000 7fa96a04a640 10 monclient: get_monmap_and_config 2024-03-29T11:37:46.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.029+0000 7fa96a04a640 10 monclient: build_initial_monmap 2024-03-29T11:37:46.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.029+0000 7fa96a04a640 10 monclient: monmap: 2024-03-29T11:37:46.029 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:46.030 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:46.030 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:37:46.030587+0000 2024-03-29T11:37:46.030 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:37:46.030587+0000 2024-03-29T11:37:46.030 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:46.030 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:46.030 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:46.030 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:37:46.030 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:37:46.030 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:46.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.030+0000 7fa96a04a640 1 Processor -- start 2024-03-29T11:37:46.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.031+0000 7fa96a04a640 1 -- start start 2024-03-29T11:37:46.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.031+0000 7fa96a04a640 10 monclient: init 2024-03-29T11:37:46.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa96a04a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:46.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa96a04a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:46.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa96a04a640 10 monclient(hunting): picked mon.noname-c con 0x7fa9640b1f60 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:46.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa96a04a640 10 monclient(hunting): picked mon.noname-b con 0x7fa964071eb0 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:46.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa96a04a640 10 monclient(hunting): picked mon.noname-a con 0x7fa964143d10 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:46.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa96a04a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96406ef80 con 0x7fa964143d10 2024-03-29T11:37:46.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa96a04a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96406f260 con 0x7fa9640b1f60 2024-03-29T11:37:46.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa96a04a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9640b40c0 con 0x7fa964071eb0 2024-03-29T11:37:46.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa96a04a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:46.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa96a04a640 10 monclient(hunting): authenticate will time out at 1048.159011s 2024-03-29T11:37:46.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa962ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa964071eb0 0x7fa964146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:56174/0 (socket says 172.21.15.155:56174) 2024-03-29T11:37:46.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.032+0000 7fa962ffd640 1 -- 172.21.15.155:0/4031367401 learned_addr learned my addr 172.21.15.155:0/4031367401 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:46.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.033+0000 7fa9627fc640 1 -- 172.21.15.155:0/4031367401 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258998986 0 0) 0x7fa9640b40c0 con 0x7fa964071eb0 2024-03-29T11:37:46.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.033+0000 7fa9627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:46.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.033+0000 7fa9627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:46.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.033+0000 7fa9627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:46.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.033+0000 7fa9627fc640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:37:46.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.033+0000 7fa9627fc640 1 -- 172.21.15.155:0/4031367401 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa94c003520 con 0x7fa964071eb0 2024-03-29T11:37:46.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.034+0000 7fa9627fc640 1 -- 172.21.15.155:0/4031367401 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2345772108 0 0) 0x7fa94c003520 con 0x7fa964071eb0 2024-03-29T11:37:46.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.034+0000 7fa9627fc640 1 -- 172.21.15.155:0/4031367401 >> v1:172.21.15.183:6789/0 conn(0x7fa9640b1f60 legacy=0x7fa964143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:46.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.035+0000 7fa9627fc640 1 -- 172.21.15.155:0/4031367401 >> v1:172.21.15.155:6789/0 conn(0x7fa964143d10 legacy=0x7fa964146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:46.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:46.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 1 monclient: found mon.noname-b 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 1 -- 172.21.15.155:0/4031367401 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa964155d70 con 0x7fa964071eb0 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 1 -- 172.21.15.155:0/4031367401 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa958001690 con 0x7fa964071eb0 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.036+0000 7fa9627fc640 10 monclient: dump: 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:46.038 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.037+0000 7fa9627fc640 1 -- 172.21.15.155:0/4031367401 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa958001920 con 0x7fa964071eb0 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.037+0000 7fa9627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.037+0000 7fa9627fc640 1 -- 172.21.15.155:0/4031367401 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa958001cf0 con 0x7fa964071eb0 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.037+0000 7fa9627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.037+0000 7fa9627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.037+0000 7fa9627fc640 10 monclient: dump: 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:46.039 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:46.040 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:46.040 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:46.040 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:46.040 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:46.040 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:46.040 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:46.040 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:46.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.038+0000 7fa96a04a640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:37:46.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.038+0000 7fa96a04a640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:46.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.038+0000 7fa96a04a640 4 set_mon_vals no callback set 2024-03-29T11:37:46.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.040+0000 7fa96a04a640 10 monclient: shutdown 2024-03-29T11:37:46.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.040+0000 7fa96a04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:46.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.040+0000 7fa96a04a640 1 -- 172.21.15.155:0/4031367401 >> v1:172.21.15.155:6790/0 conn(0x7fa964071eb0 legacy=0x7fa964146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:46.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.040+0000 7fa96a04a640 1 -- 172.21.15.155:0/4031367401 shutdown_connections 2024-03-29T11:37:46.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.040+0000 7fa96a04a640 1 -- 172.21.15.155:0/4031367401 >> 172.21.15.155:0/4031367401 conn(0x7fa964066a00 msgr2=0x7fa964066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:46.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.041+0000 7fa96a04a640 1 -- 172.21.15.155:0/4031367401 shutdown_connections 2024-03-29T11:37:46.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.041+0000 7fa96a04a640 1 -- 172.21.15.155:0/4031367401 wait complete. 2024-03-29T11:37:46.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.042+0000 7fa96a04a640 10 monclient: build_initial_monmap 2024-03-29T11:37:46.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.042+0000 7fa96a04a640 10 monclient: monmap: 2024-03-29T11:37:46.043 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:46.043 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:46.043 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:37:46.043 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:37:46.043 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:46.043 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:46.043 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:46.043 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:37:46.043 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:37:46.043 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:46.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.043+0000 7fa96a04a640 1 librados: starting msgr at 2024-03-29T11:37:46.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.043+0000 7fa96a04a640 1 librados: starting objecter 2024-03-29T11:37:46.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.043+0000 7fa96a04a640 1 Processor -- start 2024-03-29T11:37:46.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.044+0000 7fa96a04a640 1 -- start start 2024-03-29T11:37:46.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.044+0000 7fa96a04a640 1 librados: setting wanted keys 2024-03-29T11:37:46.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.044+0000 7fa96a04a640 1 librados: calling monclient init 2024-03-29T11:37:46.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.044+0000 7fa96a04a640 10 monclient: init 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.044+0000 7fa96a04a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.044+0000 7fa96a04a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.044+0000 7fa96a04a640 10 monclient(hunting): picked mon.noname-b con 0x7fa9640b1f60 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.045+0000 7fa96a04a640 10 monclient(hunting): picked mon.noname-c con 0x7fa964143d10 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.045+0000 7fa96a04a640 10 monclient(hunting): picked mon.noname-a con 0x7fa964071460 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.045+0000 7fa96a04a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa964138150 con 0x7fa964071460 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.045+0000 7fa962ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa964143d10 0x7fa964131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33382/0 (socket says 172.21.15.155:33382) 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.045+0000 7fa962ffd640 1 -- 172.21.15.155:0/2647284775 learned_addr learned my addr 172.21.15.155:0/2647284775 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.045+0000 7fa94bfff640 10 client.?.objecter ms_handle_connect 0x7fa964143d10 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.045+0000 7fa94bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.045+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa964142010 con 0x7fa9640b1f60 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.046+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa964132740 con 0x7fa964143d10 2024-03-29T11:37:46.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.046+0000 7fa96a04a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:46.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.046+0000 7fa96a04a640 10 monclient(hunting): authenticate will time out at 1048.173110s 2024-03-29T11:37:46.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.046+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557237465 0 0) 0x7fa964132740 con 0x7fa964143d10 2024-03-29T11:37:46.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.046+0000 7fa94bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:46.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.046+0000 7fa94bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:46.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.046+0000 7fa94bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:46.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.046+0000 7fa94bfff640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:37:46.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.046+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa950004eb0 con 0x7fa964143d10 2024-03-29T11:37:46.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.047+0000 7fa94bfff640 10 client.?.objecter ms_handle_connect 0x7fa9640b1f60 2024-03-29T11:37:46.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.047+0000 7fa94bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:46.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.049+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047260853 0 0) 0x7fa964142010 con 0x7fa9640b1f60 2024-03-29T11:37:46.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.049+0000 7fa94bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:46.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.049+0000 7fa94bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:46.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.049+0000 7fa94bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:46.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.049+0000 7fa94bfff640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:37:46.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.049+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa964132740 con 0x7fa9640b1f60 2024-03-29T11:37:46.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.050+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3127292584 0 0) 0x7fa964132740 con 0x7fa9640b1f60 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.050+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa964142010 con 0x7fa9640b1f60 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.050+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa954001830 con 0x7fa9640b1f60 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.050+0000 7fa94bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient(hunting): dump: 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:46.055 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042792398 0 0) 0x7fa950004eb0 con 0x7fa964143d10 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa964132740 con 0x7fa964143d10 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa958001550 con 0x7fa964143d10 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient(hunting): dump: 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:46.056 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 client.?.objecter ms_handle_connect 0x7fa964071460 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557108972 0 0) 0x7fa964138150 con 0x7fa964071460 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa950004eb0 con 0x7fa964071460 2024-03-29T11:37:46.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1512722984 0 0) 0x7fa964132740 con 0x7fa964143d10 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 >> v1:172.21.15.183:6789/0 conn(0x7fa9640b1f60 legacy=0x7fa964137a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 >> v1:172.21.15.155:6789/0 conn(0x7fa964071460 legacy=0x7fa964132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 1 monclient: found mon.c 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa964142280 con 0x7fa964143d10 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient: _finish_auth 0 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa94bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa96a04a640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa96a04a640 10 monclient: _renew_subs 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa96a04a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.051+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa964142560 con 0x7fa964143d10 2024-03-29T11:37:46.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa958001790 con 0x7fa964143d10 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa94bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa958004e10 con 0x7fa964143d10 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa94bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa94bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa94bfff640 10 monclient: dump: 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:46.059 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:46.060 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:46.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa96a04a640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:46.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa96a04a640 10 monclient: _renew_subs 2024-03-29T11:37:46.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa96a04a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:37:46.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa964141300 con 0x7fa964143d10 2024-03-29T11:37:46.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa96a04a640 1 librados: init done 2024-03-29T11:37:46.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.052+0000 7fa961ffb640 4 set_mon_vals no callback set 2024-03-29T11:37:46.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.054+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fa95801ade0 con 0x7fa964143d10 2024-03-29T11:37:46.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.054+0000 7fa96a04a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:46.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.054+0000 7fa96a04a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:46.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.054+0000 7fa96a04a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:37:46.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.054+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa944005080 con 0x7fa964143d10 2024-03-29T11:37:46.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.056+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fa958084b70 con 0x7fa964143d10 2024-03-29T11:37:46.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.056+0000 7fa94bfff640 10 client.4343.objecter ms_dispatch 0x7fa964000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:46.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.056+0000 7fa94bfff640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:46.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.056+0000 7fa94bfff640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:46.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.056+0000 7fa9637fe640 10 monclient: get_auth_request con 0x7fa95006dfd0 auth_method 0 2024-03-29T11:37:46.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.056+0000 7fa94bfff640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T11:37:46.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.056+0000 7fa94bfff640 10 client.4343.objecter ms_handle_connect 0x7fa95006dfd0 2024-03-29T11:37:46.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.063+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa958084e60 con 0x7fa964143d10 2024-03-29T11:37:46.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.063+0000 7fa94bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:46.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.063+0000 7fa94bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:46.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.386+0000 7fa96a04a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:46.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.386+0000 7fa96a04a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:46.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.386+0000 7fa96a04a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:37:46.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.386+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa944005330 con 0x7fa964143d10 2024-03-29T11:37:46.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.388+0000 7fa94bfff640 1 -- 172.21.15.155:0/2647284775 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fa958050450 con 0x7fa964143d10 2024-03-29T11:37:46.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.388+0000 7fa94bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:46.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.388+0000 7fa94bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:46.389 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:37:46.389 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:46.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.392+0000 7fa96a04a640 10 librados: watch_flush enter 2024-03-29T11:37:46.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.393+0000 7fa96a04a640 10 librados: watch_flush exit 2024-03-29T11:37:46.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.393+0000 7fa96a04a640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T11:37:46.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.393+0000 7fa96a04a640 10 client.4343.objecter successfully canceled tick 2024-03-29T11:37:46.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.395+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 >> v1:172.21.15.155:6816/1760814708 conn(0x7fa95006dfd0 legacy=0x7fa950070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:46.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.395+0000 7fa96a04a640 10 monclient: shutdown 2024-03-29T11:37:46.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.395+0000 7fa96a04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:46.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.395+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 >> v1:172.21.15.155:6790/0 conn(0x7fa964143d10 legacy=0x7fa964131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:46.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.397+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 shutdown_connections 2024-03-29T11:37:46.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.397+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 >> 172.21.15.155:0/2647284775 conn(0x7fa964066a00 msgr2=0x7fa9640b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:46.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.398+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 shutdown_connections 2024-03-29T11:37:46.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.398+0000 7fa96a04a640 1 -- 172.21.15.155:0/2647284775 wait complete. 2024-03-29T11:37:46.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:46.398+0000 7fa96a04a640 1 librados: shutdown 2024-03-29T11:37:46.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:53.222 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:53.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.367+0000 7fc0a9946640 10 monclient: get_monmap_and_config 2024-03-29T11:37:53.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.367+0000 7fc0a9946640 10 monclient: build_initial_monmap 2024-03-29T11:37:53.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.367+0000 7fc0a9946640 10 monclient: monmap: 2024-03-29T11:37:53.368 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:53.368 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:53.368 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:37:53.368851+0000 2024-03-29T11:37:53.368 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:37:53.368851+0000 2024-03-29T11:37:53.368 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:53.368 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:53.369 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:53.369 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:37:53.369 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:37:53.369 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:53.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.368+0000 7fc0a9946640 1 Processor -- start 2024-03-29T11:37:53.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.368+0000 7fc0a9946640 1 -- start start 2024-03-29T11:37:53.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.368+0000 7fc0a9946640 10 monclient: init 2024-03-29T11:37:53.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a9946640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:53.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a9946640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:53.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a9946640 10 monclient(hunting): picked mon.noname-c con 0x7fc0a41457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:53.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a9946640 10 monclient(hunting): picked mon.noname-b con 0x7fc0a413f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:53.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a9946640 10 monclient(hunting): picked mon.noname-a con 0x7fc0a40a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:53.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a9946640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a40aaef0 con 0x7fc0a40a9d70 2024-03-29T11:37:53.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a9946640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a4145bd0 con 0x7fc0a41457f0 2024-03-29T11:37:53.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a9946640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a4145eb0 con 0x7fc0a413f650 2024-03-29T11:37:53.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a9946640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:53.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a9946640 10 monclient(hunting): authenticate will time out at 1055.496206s 2024-03-29T11:37:53.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a2ffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fc0a41457f0 0x7fc0a413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:46446/0 (socket says 172.21.15.155:46446) 2024-03-29T11:37:53.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.369+0000 7fc0a2ffd640 1 -- 172.21.15.155:0/3384415103 learned_addr learned my addr 172.21.15.155:0/3384415103 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:53.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732886660 0 0) 0x7fc0a40aaef0 con 0x7fc0a40a9d70 2024-03-29T11:37:53.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:53.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:53.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc08c003520 con 0x7fc0a40a9d70 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516940502 0 0) 0x7fc0a4145eb0 con 0x7fc0a413f650 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a40aaef0 con 0x7fc0a413f650 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144045977 0 0) 0x7fc0a4145bd0 con 0x7fc0a41457f0 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:37:53.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a4145eb0 con 0x7fc0a41457f0 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3593691246 0 0) 0x7fc0a40aaef0 con 0x7fc0a413f650 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.370+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 >> v1:172.21.15.183:6789/0 conn(0x7fc0a41457f0 legacy=0x7fc0a413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 >> v1:172.21.15.155:6789/0 conn(0x7fc0a40a9d70 legacy=0x7fc0a414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a4146340 con 0x7fc0a413f650 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a9946640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a9946640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc0980015f0 con 0x7fc0a413f650 2024-03-29T11:37:53.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 10 monclient: dump: 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:53.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a9946640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0980017d0 con 0x7fc0a413f650 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a9946640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a9946640 4 set_mon_vals no callback set 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 1 -- 172.21.15.155:0/3384415103 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc098001be0 con 0x7fc0a413f650 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.371+0000 7fc0a1ffb640 10 monclient: dump: 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:53.377 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.372+0000 7fc0a9946640 10 monclient: shutdown 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.372+0000 7fc0a9946640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.372+0000 7fc0a9946640 1 -- 172.21.15.155:0/3384415103 >> v1:172.21.15.155:6790/0 conn(0x7fc0a413f650 legacy=0x7fc0a40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.372+0000 7fc0a9946640 1 -- 172.21.15.155:0/3384415103 shutdown_connections 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.372+0000 7fc0a9946640 1 -- 172.21.15.155:0/3384415103 >> 172.21.15.155:0/3384415103 conn(0x7fc0a4066a00 msgr2=0x7fc0a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.372+0000 7fc0a9946640 1 -- 172.21.15.155:0/3384415103 shutdown_connections 2024-03-29T11:37:53.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.372+0000 7fc0a9946640 1 -- 172.21.15.155:0/3384415103 wait complete. 2024-03-29T11:37:53.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.373+0000 7fc0a9946640 10 monclient: build_initial_monmap 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.373+0000 7fc0a9946640 10 monclient: monmap: 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.373+0000 7fc0a9946640 1 librados: starting msgr at 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.373+0000 7fc0a9946640 1 librados: starting objecter 2024-03-29T11:37:53.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.374+0000 7fc0a9946640 1 Processor -- start 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.374+0000 7fc0a9946640 1 -- start start 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.374+0000 7fc0a9946640 1 librados: setting wanted keys 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.374+0000 7fc0a9946640 1 librados: calling monclient init 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.374+0000 7fc0a9946640 10 monclient: init 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a9946640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a9946640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a9946640 10 monclient(hunting): picked mon.noname-c con 0x7fc0a40a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a9946640 10 monclient(hunting): picked mon.noname-b con 0x7fc0a41457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a9946640 10 monclient(hunting): picked mon.noname-a con 0x7fc0a40ada60 addr v1:172.21.15.155:6789/0 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a9946640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a40ae4e0 con 0x7fc0a40ada60 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a9946640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a40adfc0 con 0x7fc0a41457f0 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a9946640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a40b1480 con 0x7fc0a40a9d70 2024-03-29T11:37:53.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a9946640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a9946640 10 monclient(hunting): authenticate will time out at 1055.502414s 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a2ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fc0a40a9d70 0x7fc0a40ad350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51062/0 (socket says 172.21.15.155:51062) 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc0a2ffd640 1 -- 172.21.15.155:0/1692213572 learned_addr learned my addr 172.21.15.155:0/1692213572 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc08bfff640 10 client.?.objecter ms_handle_connect 0x7fc0a40a9d70 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.375+0000 7fc08bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 client.?.objecter ms_handle_connect 0x7fc0a41457f0 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657156707 0 0) 0x7fc0a40adfc0 con 0x7fc0a41457f0 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc090004eb0 con 0x7fc0a41457f0 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869166314 0 0) 0x7fc0a40b1480 con 0x7fc0a40a9d70 2024-03-29T11:37:53.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a40adfc0 con 0x7fc0a40a9d70 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 client.?.objecter ms_handle_connect 0x7fc0a40ada60 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 311408512 0 0) 0x7fc090004eb0 con 0x7fc0a41457f0 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0a40b1480 con 0x7fc0a41457f0 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc098001ae0 con 0x7fc0a41457f0 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.376+0000 7fc08bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 10 monclient(hunting): dump: 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:53.390 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539327426 0 0) 0x7fc0a40ae4e0 con 0x7fc0a40ada60 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:37:53.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc090004eb0 con 0x7fc0a40ada60 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 566962858 0 0) 0x7fc0a40b1480 con 0x7fc0a41457f0 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 >> v1:172.21.15.155:6790/0 conn(0x7fc0a40a9d70 legacy=0x7fc0a40ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 >> v1:172.21.15.155:6789/0 conn(0x7fc0a40ada60 legacy=0x7fc0a41d7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 1 monclient: found mon.b 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a40b1610 con 0x7fc0a41457f0 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 10 monclient: _finish_auth 0 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc08bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc0a9946640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc0a9946640 10 monclient: _renew_subs 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc0a9946640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:37:53.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc0a9946640 1 -- 172.21.15.155:0/1692213572 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a4142160 con 0x7fc0a41457f0 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc0a9946640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc0a9946640 10 monclient: _renew_subs 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc0a9946640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc0a9946640 1 -- 172.21.15.155:0/1692213572 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a41d7a60 con 0x7fc0a41457f0 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.377+0000 7fc0a9946640 1 librados: init done 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.378+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0980038a0 con 0x7fc0a41457f0 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.378+0000 7fc08bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.378+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc098003bb0 con 0x7fc0a41457f0 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.378+0000 7fc08bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.378+0000 7fc08bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.378+0000 7fc0a17fa640 4 set_mon_vals no callback set 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.378+0000 7fc08bfff640 10 monclient: dump: 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:37:53.393 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.378+0000 7fc0a9946640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.379+0000 7fc0a9946640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.379+0000 7fc0a9946640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.379+0000 7fc0a9946640 1 -- 172.21.15.155:0/1692213572 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc084005080 con 0x7fc0a41457f0 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.380+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fc098019b80 con 0x7fc0a41457f0 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.381+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fc098083640 con 0x7fc0a41457f0 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.381+0000 7fc08bfff640 10 client.4348.objecter ms_dispatch 0x7fc0a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:53.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.381+0000 7fc08bfff640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:53.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.381+0000 7fc08bfff640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:53.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.381+0000 7fc08bfff640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T11:37:53.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.381+0000 7fc0a2ffd640 10 monclient: get_auth_request con 0x7fc09006dff0 auth_method 0 2024-03-29T11:37:53.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.382+0000 7fc08bfff640 10 client.4348.objecter ms_handle_connect 0x7fc09006dff0 2024-03-29T11:37:53.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.387+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc09804ef80 con 0x7fc0a41457f0 2024-03-29T11:37:53.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.387+0000 7fc08bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:53.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.387+0000 7fc08bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:53.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.766+0000 7fc0a9946640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:53.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.766+0000 7fc0a9946640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:53.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.766+0000 7fc0a9946640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:37:53.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.766+0000 7fc0a9946640 1 -- 172.21.15.155:0/1692213572 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc084005330 con 0x7fc0a41457f0 2024-03-29T11:37:53.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.767+0000 7fc08bfff640 1 -- 172.21.15.155:0/1692213572 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fc098003a10 con 0x7fc0a41457f0 2024-03-29T11:37:53.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.767+0000 7fc08bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:53.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.767+0000 7fc08bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:53.768 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:37:53.768 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:53.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.771+0000 7fc0a9946640 10 librados: watch_flush enter 2024-03-29T11:37:53.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.771+0000 7fc0a9946640 10 librados: watch_flush exit 2024-03-29T11:37:53.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.771+0000 7fc0a9946640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T11:37:53.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.771+0000 7fc0a9946640 10 client.4348.objecter successfully canceled tick 2024-03-29T11:37:53.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.771+0000 7fc0a9946640 1 -- 172.21.15.155:0/1692213572 >> v1:172.21.15.155:6816/1760814708 conn(0x7fc09006dff0 legacy=0x7fc0900704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.771+0000 7fc0a9946640 10 monclient: shutdown 2024-03-29T11:37:53.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.771+0000 7fc0a9946640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:53.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.771+0000 7fc0a9946640 1 -- 172.21.15.155:0/1692213572 >> v1:172.21.15.183:6789/0 conn(0x7fc0a41457f0 legacy=0x7fc0a40b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.771+0000 7fc0a9946640 1 -- 172.21.15.155:0/1692213572 shutdown_connections 2024-03-29T11:37:53.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.771+0000 7fc0a9946640 1 -- 172.21.15.155:0/1692213572 >> 172.21.15.155:0/1692213572 conn(0x7fc0a4066a00 msgr2=0x7fc0a4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:53.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.771+0000 7fc0a9946640 1 -- 172.21.15.155:0/1692213572 shutdown_connections 2024-03-29T11:37:53.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.772+0000 7fc0a9946640 1 -- 172.21.15.155:0/1692213572 wait complete. 2024-03-29T11:37:53.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:37:53.772+0000 7fc0a9946640 1 librados: shutdown 2024-03-29T11:37:53.786 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:00.593 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:00.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.740+0000 7f123cfc8640 10 monclient: get_monmap_and_config 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.740+0000 7f123cfc8640 10 monclient: build_initial_monmap 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.740+0000 7f123cfc8640 10 monclient: monmap: 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:38:00.740510+0000 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:38:00.740510+0000 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:38:00.740 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:00.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.741+0000 7f123cfc8640 1 Processor -- start 2024-03-29T11:38:00.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.741+0000 7f123cfc8640 1 -- start start 2024-03-29T11:38:00.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.741+0000 7f123cfc8640 10 monclient: init 2024-03-29T11:38:00.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.743+0000 7f123cfc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:00.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.743+0000 7f123cfc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:00.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.743+0000 7f123cfc8640 10 monclient(hunting): picked mon.noname-a con 0x7f12381457f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:00.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.743+0000 7f123cfc8640 10 monclient(hunting): picked mon.noname-b con 0x7f123813f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:00.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.743+0000 7f123cfc8640 10 monclient(hunting): picked mon.noname-c con 0x7f12380a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:00.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.743+0000 7f123cfc8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12380aaef0 con 0x7f12381457f0 2024-03-29T11:38:00.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.743+0000 7f123cfc8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1238145bd0 con 0x7f12380a9d70 2024-03-29T11:38:00.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.743+0000 7f123cfc8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1238145eb0 con 0x7f123813f650 2024-03-29T11:38:00.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.744+0000 7f1235d9b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f123813f650 0x7f12380b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:35622/0 (socket says 172.21.15.155:35622) 2024-03-29T11:38:00.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.744+0000 7f1235d9b640 1 -- 172.21.15.155:0/725263487 learned_addr learned my addr 172.21.15.155:0/725263487 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:00.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 1 -- 172.21.15.155:0/725263487 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199481131 0 0) 0x7f12380aaef0 con 0x7f12381457f0 2024-03-29T11:38:00.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123cfc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:00.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123cfc8640 10 monclient(hunting): authenticate will time out at 1062.871173s 2024-03-29T11:38:00.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:00.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:00.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:00.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:38:00.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 1 -- 172.21.15.155:0/725263487 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1220003520 con 0x7f12381457f0 2024-03-29T11:38:00.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 1 -- 172.21.15.155:0/725263487 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145539132 0 0) 0x7f1238145eb0 con 0x7f123813f650 2024-03-29T11:38:00.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.745+0000 7f123559a640 1 -- 172.21.15.155:0/725263487 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12380aaef0 con 0x7f123813f650 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 1 -- 172.21.15.155:0/725263487 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2279157351 0 0) 0x7f1220003520 con 0x7f12381457f0 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 1 -- 172.21.15.155:0/725263487 >> v1:172.21.15.155:6790/0 conn(0x7f123813f650 legacy=0x7f12380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 1 -- 172.21.15.155:0/725263487 >> v1:172.21.15.183:6789/0 conn(0x7f12380a9d70 legacy=0x7f123814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 1 monclient: found mon.noname-a 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 1 -- 172.21.15.155:0/725263487 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1238146340 con 0x7f12381457f0 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 10 monclient: _finish_auth 0 2024-03-29T11:38:00.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 1 -- 172.21.15.155:0/725263487 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f122c001540 con 0x7f12381457f0 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123cfc8640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123cfc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 10 monclient: dump: 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:00.748 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123cfc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 1 -- 172.21.15.155:0/725263487 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f122c0016a0 con 0x7f12381457f0 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123559a640 1 -- 172.21.15.155:0/725263487 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f122c001ab0 con 0x7f12381457f0 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.746+0000 7f123cfc8640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.747+0000 7f123cfc8640 4 set_mon_vals no callback set 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.747+0000 7f123559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.747+0000 7f123559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.747+0000 7f123559a640 10 monclient: dump: 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:00.749 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.750 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.750 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:00.750 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:00.750 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:00.750 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:00.750 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:00.750 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:00.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.748+0000 7f123cfc8640 10 monclient: shutdown 2024-03-29T11:38:00.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.748+0000 7f123cfc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:00.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.748+0000 7f123cfc8640 1 -- 172.21.15.155:0/725263487 >> v1:172.21.15.155:6789/0 conn(0x7f12381457f0 legacy=0x7f123813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:00.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.748+0000 7f123cfc8640 1 -- 172.21.15.155:0/725263487 shutdown_connections 2024-03-29T11:38:00.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.748+0000 7f123cfc8640 1 -- 172.21.15.155:0/725263487 >> 172.21.15.155:0/725263487 conn(0x7f1238066a00 msgr2=0x7f1238066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.749+0000 7f123cfc8640 1 -- 172.21.15.155:0/725263487 shutdown_connections 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.749+0000 7f123cfc8640 1 -- 172.21.15.155:0/725263487 wait complete. 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.751+0000 7f123cfc8640 10 monclient: build_initial_monmap 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.751+0000 7f123cfc8640 10 monclient: monmap: 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:38:00.752 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:00.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.751+0000 7f123cfc8640 1 librados: starting msgr at 2024-03-29T11:38:00.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.751+0000 7f123cfc8640 1 librados: starting objecter 2024-03-29T11:38:00.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.751+0000 7f123cfc8640 1 Processor -- start 2024-03-29T11:38:00.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.751+0000 7f123cfc8640 1 -- start start 2024-03-29T11:38:00.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.751+0000 7f123cfc8640 1 librados: setting wanted keys 2024-03-29T11:38:00.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.751+0000 7f123cfc8640 1 librados: calling monclient init 2024-03-29T11:38:00.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.751+0000 7f123cfc8640 10 monclient: init 2024-03-29T11:38:00.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.752+0000 7f123cfc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.752+0000 7f123cfc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.752+0000 7f123cfc8640 10 monclient(hunting): picked mon.noname-a con 0x7f12380a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f123cfc8640 10 monclient(hunting): picked mon.noname-c con 0x7f123813f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f123659c640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f12380a9d70 0x7f1238141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:53896/0 (socket says 172.21.15.155:53896) 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f1235d9b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f123813f650 0x7f1238137170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36740/0 (socket says 172.21.15.155:36740) 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f123659c640 1 -- 172.21.15.155:0/361428557 learned_addr learned my addr 172.21.15.155:0/361428557 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f123cfc8640 10 monclient(hunting): picked mon.noname-b con 0x7f12381457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f1226ffd640 10 client.?.objecter ms_handle_connect 0x7f123813f650 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f1226ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f1226ffd640 10 client.?.objecter ms_handle_connect 0x7f12380a9d70 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f1226ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123806e7c0 con 0x7f12380a9d70 2024-03-29T11:38:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12381404c0 con 0x7f12381457f0 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1238137f90 con 0x7f123813f650 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f123cfc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.753+0000 7f123cfc8640 10 monclient(hunting): authenticate will time out at 1062.879555s 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 client.?.objecter ms_handle_connect 0x7f12381457f0 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394425485 0 0) 0x7f1238137f90 con 0x7f123813f650 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1218004eb0 con 0x7f123813f650 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719884978 0 0) 0x7f123806e7c0 con 0x7f12380a9d70 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1238137f90 con 0x7f12380a9d70 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070007094 0 0) 0x7f12381404c0 con 0x7f12381457f0 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.754+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123806e7c0 con 0x7f12381457f0 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2105338432 0 0) 0x7f1218004eb0 con 0x7f123813f650 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12381404c0 con 0x7f123813f650 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f12280018f0 con 0x7f123813f650 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1033718394 0 0) 0x7f123806e7c0 con 0x7f12381457f0 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1218004eb0 con 0x7f12381457f0 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 22859765 0 0) 0x7f1238137f90 con 0x7f12380a9d70 2024-03-29T11:38:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f123806e7c0 con 0x7f12380a9d70 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1230001a30 con 0x7f12381457f0 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.755+0000 7f1226ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:00.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f122c0013a0 con 0x7f12380a9d70 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:00.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1210190172 0 0) 0x7f1218004eb0 con 0x7f12381457f0 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 >> v1:172.21.15.155:6790/0 conn(0x7f123813f650 legacy=0x7f1238137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 >> v1:172.21.15.155:6789/0 conn(0x7f12380a9d70 legacy=0x7f1238141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 1 monclient: found mon.b 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123806e9f0 con 0x7f12381457f0 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f1226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f123cfc8640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f123cfc8640 10 monclient: _renew_subs 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f123cfc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123806ecd0 con 0x7f12381457f0 2024-03-29T11:38:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f123cfc8640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f123cfc8640 10 monclient: _renew_subs 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f123cfc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1238140950 con 0x7f12381457f0 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.756+0000 7f123cfc8640 1 librados: init done 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.757+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1230001b90 con 0x7f12381457f0 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.757+0000 7f1226ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.757+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1230003970 con 0x7f12381457f0 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.757+0000 7f1227fff640 4 set_mon_vals no callback set 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.757+0000 7f1226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.757+0000 7f1226ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.757+0000 7f1226ffd640 10 monclient: dump: 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.768 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.758+0000 7f123cfc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.758+0000 7f123cfc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.758+0000 7f123cfc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.758+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121c005080 con 0x7f12381457f0 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.759+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f1230019940 con 0x7f12381457f0 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.760+0000 7f123659c640 10 monclient: get_auth_request con 0x7f121806e1e0 auth_method 0 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.760+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f1230083980 con 0x7f12381457f0 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.760+0000 7f1226ffd640 10 client.4351.objecter ms_dispatch 0x7f1238000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:00.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.760+0000 7f1226ffd640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:00.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.760+0000 7f1226ffd640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:00.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.761+0000 7f1226ffd640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-29T11:38:00.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.761+0000 7f1226ffd640 10 client.4351.objecter ms_handle_connect 0x7f121806e1e0 2024-03-29T11:38:00.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.769+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f123004f090 con 0x7f12381457f0 2024-03-29T11:38:00.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.769+0000 7f1226ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:00.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:00.769+0000 7f1226ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:01.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.134+0000 7f123cfc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:01.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.135+0000 7f123cfc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:01.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.135+0000 7f123cfc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:01.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.135+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f121c005330 con 0x7f12381457f0 2024-03-29T11:38:01.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.135+0000 7f1226ffd640 1 -- 172.21.15.155:0/361428557 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f1230051190 con 0x7f12381457f0 2024-03-29T11:38:01.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.135+0000 7f1226ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:01.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.135+0000 7f1226ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:01.135 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:38:01.136 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:01.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 10 librados: watch_flush enter 2024-03-29T11:38:01.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 10 librados: watch_flush exit 2024-03-29T11:38:01.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-29T11:38:01.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 10 client.4351.objecter successfully canceled tick 2024-03-29T11:38:01.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 >> v1:172.21.15.155:6816/1760814708 conn(0x7f121806e1e0 legacy=0x7f12180706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:01.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 10 monclient: shutdown 2024-03-29T11:38:01.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:01.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 >> v1:172.21.15.183:6789/0 conn(0x7f12381457f0 legacy=0x7f1238137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:01.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 shutdown_connections 2024-03-29T11:38:01.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 >> 172.21.15.155:0/361428557 conn(0x7f1238066a00 msgr2=0x7f12380b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:01.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 shutdown_connections 2024-03-29T11:38:01.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.139+0000 7f123cfc8640 1 -- 172.21.15.155:0/361428557 wait complete. 2024-03-29T11:38:01.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:01.140+0000 7f123cfc8640 1 librados: shutdown 2024-03-29T11:38:01.153 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:07.959 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:08.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.113+0000 7fc40e24c640 10 monclient: get_monmap_and_config 2024-03-29T11:38:08.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.113+0000 7fc40e24c640 10 monclient: build_initial_monmap 2024-03-29T11:38:08.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.113+0000 7fc40e24c640 10 monclient: monmap: 2024-03-29T11:38:08.116 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:08.116 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:08.116 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:38:08.113845+0000 2024-03-29T11:38:08.116 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:38:08.113845+0000 2024-03-29T11:38:08.116 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:08.116 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:08.117 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:08.117 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:38:08.117 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:38:08.117 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:08.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.115+0000 7fc40e24c640 1 Processor -- start 2024-03-29T11:38:08.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.115+0000 7fc40e24c640 1 -- start start 2024-03-29T11:38:08.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.115+0000 7fc40e24c640 10 monclient: init 2024-03-29T11:38:08.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.116+0000 7fc40e24c640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:08.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.116+0000 7fc40e24c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:08.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.116+0000 7fc40e24c640 10 monclient(hunting): picked mon.noname-a con 0x7fc400006e30 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:08.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.116+0000 7fc40e24c640 10 monclient(hunting): picked mon.noname-b con 0x7fc400035e00 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:08.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.116+0000 7fc40e24c640 10 monclient(hunting): picked mon.noname-c con 0x7fc4000c9670 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:08.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.116+0000 7fc40e24c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc400035970 con 0x7fc400006e30 2024-03-29T11:38:08.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.116+0000 7fc40e24c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc400036280 con 0x7fc4000c9670 2024-03-29T11:38:08.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.116+0000 7fc40e24c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc400035640 con 0x7fc400035e00 2024-03-29T11:38:08.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.116+0000 7fc40e24c640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:08.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.116+0000 7fc40e24c640 10 monclient(hunting): authenticate will time out at 1070.242543s 2024-03-29T11:38:08.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.119+0000 7fc40da4b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fc4000c9670 0x7fc4000d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:40357/0 (socket says 172.21.15.155:40357) 2024-03-29T11:38:08.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.119+0000 7fc40da4b640 1 -- 172.21.15.155:0/774786102 learned_addr learned my addr 172.21.15.155:0/774786102 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:08.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 1 -- 172.21.15.155:0/774786102 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71411118 0 0) 0x7fc400036280 con 0x7fc4000c9670 2024-03-29T11:38:08.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:08.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:08.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:08.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:38:08.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 1 -- 172.21.15.155:0/774786102 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f4003520 con 0x7fc4000c9670 2024-03-29T11:38:08.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 1 -- 172.21.15.155:0/774786102 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165365790 0 0) 0x7fc400035640 con 0x7fc400035e00 2024-03-29T11:38:08.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:08.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:08.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:08.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:38:08.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 1 -- 172.21.15.155:0/774786102 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f4003840 con 0x7fc400035e00 2024-03-29T11:38:08.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 1 -- 172.21.15.155:0/774786102 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1802739715 0 0) 0x7fc3f4003520 con 0x7fc4000c9670 2024-03-29T11:38:08.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 1 -- 172.21.15.155:0/774786102 >> v1:172.21.15.155:6790/0 conn(0x7fc400035e00 legacy=0x7fc4000c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:08.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.120+0000 7fc3fffff640 1 -- 172.21.15.155:0/774786102 >> v1:172.21.15.155:6789/0 conn(0x7fc400006e30 legacy=0x7fc4000cbcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:08.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:08.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 1 monclient: found mon.noname-c 2024-03-29T11:38:08.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:08.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:38:08.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 1 -- 172.21.15.155:0/774786102 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc400034e20 con 0x7fc4000c9670 2024-03-29T11:38:08.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 10 monclient: _finish_auth 0 2024-03-29T11:38:08.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:08.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:08.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 1 -- 172.21.15.155:0/774786102 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc404001670 con 0x7fc4000c9670 2024-03-29T11:38:08.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:08.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 10 monclient: dump: 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc40e24c640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc40e24c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:08.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 1 -- 172.21.15.155:0/774786102 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc404001850 con 0x7fc4000c9670 2024-03-29T11:38:08.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc3fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:08.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc40e24c640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:08.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.121+0000 7fc40e24c640 4 set_mon_vals no callback set 2024-03-29T11:38:08.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.122+0000 7fc40e24c640 10 monclient: shutdown 2024-03-29T11:38:08.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.122+0000 7fc40e24c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:08.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.122+0000 7fc40e24c640 1 -- 172.21.15.155:0/774786102 >> v1:172.21.15.183:6789/0 conn(0x7fc4000c9670 legacy=0x7fc4000d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:08.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.123+0000 7fc3fffff640 1 -- 172.21.15.155:0/774786102 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc404001c60 con 0x7fc4000c9670 2024-03-29T11:38:08.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.123+0000 7fc3fffff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:38:08.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.124+0000 7fc40e24c640 1 -- 172.21.15.155:0/774786102 shutdown_connections 2024-03-29T11:38:08.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.124+0000 7fc40e24c640 1 -- 172.21.15.155:0/774786102 >> 172.21.15.155:0/774786102 conn(0x7fc400005c00 msgr2=0x7fc400006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:08.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.125+0000 7fc40e24c640 1 -- 172.21.15.155:0/774786102 shutdown_connections 2024-03-29T11:38:08.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.127+0000 7fc40e24c640 1 -- 172.21.15.155:0/774786102 wait complete. 2024-03-29T11:38:08.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.128+0000 7fc40e24c640 10 monclient: build_initial_monmap 2024-03-29T11:38:08.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.128+0000 7fc40e24c640 10 monclient: monmap: 2024-03-29T11:38:08.128 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:08.128 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:08.128 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:38:08.128 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:38:08.129 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:08.129 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:08.129 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:08.129 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:38:08.129 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:38:08.129 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:08.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.128+0000 7fc40e24c640 1 librados: starting msgr at 2024-03-29T11:38:08.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.128+0000 7fc40e24c640 1 librados: starting objecter 2024-03-29T11:38:08.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.128+0000 7fc40e24c640 1 Processor -- start 2024-03-29T11:38:08.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 1 -- start start 2024-03-29T11:38:08.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 1 librados: setting wanted keys 2024-03-29T11:38:08.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 1 librados: calling monclient init 2024-03-29T11:38:08.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 10 monclient: init 2024-03-29T11:38:08.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:08.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:08.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 10 monclient(hunting): picked mon.noname-c con 0x7fc400006e30 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:08.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 10 monclient(hunting): picked mon.noname-a con 0x7fc400035e00 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:08.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 10 monclient(hunting): picked mon.noname-b con 0x7fc4000c9670 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:08.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40002d150 con 0x7fc400035e00 2024-03-29T11:38:08.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40002ded0 con 0x7fc4000c9670 2024-03-29T11:38:08.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40002e1b0 con 0x7fc400006e30 2024-03-29T11:38:08.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:08.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.129+0000 7fc40e24c640 10 monclient(hunting): authenticate will time out at 1070.255813s 2024-03-29T11:38:08.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.133+0000 7fc40d24a640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fc400006e30 0x7fc4000bfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:40108/0 (socket says 172.21.15.155:40108) 2024-03-29T11:38:08.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.133+0000 7fc40d24a640 1 -- 172.21.15.155:0/3603242749 learned_addr learned my addr 172.21.15.155:0/3603242749 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:08.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 10 client.?.objecter ms_handle_connect 0x7fc4000c9670 2024-03-29T11:38:08.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:08.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268244447 0 0) 0x7fc40002ded0 con 0x7fc4000c9670 2024-03-29T11:38:08.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:08.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:08.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:08.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:38:08.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ec004eb0 con 0x7fc4000c9670 2024-03-29T11:38:08.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 10 client.?.objecter ms_handle_connect 0x7fc400006e30 2024-03-29T11:38:08.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:08.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1257274135 0 0) 0x7fc3ec004eb0 con 0x7fc4000c9670 2024-03-29T11:38:08.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.134+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3ec005300 con 0x7fc4000c9670 2024-03-29T11:38:08.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc404001b60 con 0x7fc4000c9670 2024-03-29T11:38:08.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:08.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:08.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient(hunting): dump: 2024-03-29T11:38:08.137 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:08.137 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:08.137 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:08.137 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:08.137 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774384914 0 0) 0x7fc40002e1b0 con 0x7fc400006e30 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ec005850 con 0x7fc400006e30 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2141340711 0 0) 0x7fc3ec005300 con 0x7fc4000c9670 2024-03-29T11:38:08.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 >> v1:172.21.15.155:6790/0 conn(0x7fc400006e30 legacy=0x7fc4000bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 >> v1:172.21.15.155:6789/0 conn(0x7fc400035e00 legacy=0x7fc400164490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 1 monclient: found mon.b 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40002e490 con 0x7fc4000c9670 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc404004b00 con 0x7fc4000c9670 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fc404004f50 con 0x7fc4000c9670 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:08.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.136+0000 7fc3fdffb640 10 monclient: dump: 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:08.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.137+0000 7fc3ff7fe640 4 set_mon_vals no callback set 2024-03-29T11:38:08.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.137+0000 7fc40e24c640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:38:08.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.137+0000 7fc40e24c640 10 monclient: _renew_subs 2024-03-29T11:38:08.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.137+0000 7fc40e24c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:08.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.137+0000 7fc40e24c640 1 -- 172.21.15.155:0/3603242749 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40002d3f0 con 0x7fc4000c9670 2024-03-29T11:38:08.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.137+0000 7fc40e24c640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:08.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.137+0000 7fc40e24c640 10 monclient: _renew_subs 2024-03-29T11:38:08.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.137+0000 7fc40e24c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:08.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.137+0000 7fc40e24c640 1 -- 172.21.15.155:0/3603242749 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4000c53c0 con 0x7fc4000c9670 2024-03-29T11:38:08.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.137+0000 7fc40e24c640 1 librados: init done 2024-03-29T11:38:08.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.139+0000 7fc40e24c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:08.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.139+0000 7fc40e24c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:08.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.139+0000 7fc40e24c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:08.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.139+0000 7fc40e24c640 1 -- 172.21.15.155:0/3603242749 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4000c53c0 con 0x7fc4000c9670 2024-03-29T11:38:08.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.142+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fc404005130 con 0x7fc4000c9670 2024-03-29T11:38:08.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.144+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fc404004d70 con 0x7fc4000c9670 2024-03-29T11:38:08.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.144+0000 7fc3fdffb640 10 client.4357.objecter ms_dispatch 0x7fc408000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:08.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.144+0000 7fc3fdffb640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:08.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.144+0000 7fc3fdffb640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:08.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.144+0000 7fc40d24a640 10 monclient: get_auth_request con 0x7fc3ec06e260 auth_method 0 2024-03-29T11:38:08.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.144+0000 7fc3fdffb640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-29T11:38:08.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.145+0000 7fc3fdffb640 10 client.4357.objecter ms_handle_connect 0x7fc3ec06e260 2024-03-29T11:38:08.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.149+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc404050a10 con 0x7fc4000c9670 2024-03-29T11:38:08.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.149+0000 7fc3fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:08.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.149+0000 7fc3fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:08.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.476+0000 7fc40e24c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:08.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.476+0000 7fc40e24c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:08.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.476+0000 7fc40e24c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:08.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.476+0000 7fc40e24c640 1 -- 172.21.15.155:0/3603242749 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4000c53c0 con 0x7fc4000c9670 2024-03-29T11:38:08.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.478+0000 7fc3fdffb640 1 -- 172.21.15.155:0/3603242749 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fc404050bb0 con 0x7fc4000c9670 2024-03-29T11:38:08.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.478+0000 7fc3fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:08.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.478+0000 7fc3fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:08.478 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:38:08.479 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:08.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.482+0000 7fc40e24c640 10 librados: watch_flush enter 2024-03-29T11:38:08.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.483+0000 7fc40e24c640 10 librados: watch_flush exit 2024-03-29T11:38:08.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.483+0000 7fc40e24c640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-29T11:38:08.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.483+0000 7fc40e24c640 10 client.4357.objecter successfully canceled tick 2024-03-29T11:38:08.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.483+0000 7fc40e24c640 1 -- 172.21.15.155:0/3603242749 >> v1:172.21.15.155:6816/1760814708 conn(0x7fc3ec06e260 legacy=0x7fc3ec070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:08.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.483+0000 7fc40e24c640 10 monclient: shutdown 2024-03-29T11:38:08.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.483+0000 7fc40e24c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:08.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.483+0000 7fc40e24c640 1 -- 172.21.15.155:0/3603242749 >> v1:172.21.15.183:6789/0 conn(0x7fc4000c9670 legacy=0x7fc400164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:08.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.484+0000 7fc40e24c640 1 -- 172.21.15.155:0/3603242749 shutdown_connections 2024-03-29T11:38:08.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.484+0000 7fc40e24c640 1 -- 172.21.15.155:0/3603242749 >> 172.21.15.155:0/3603242749 conn(0x7fc400005c00 msgr2=0x7fc4000c7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:08.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.484+0000 7fc40e24c640 1 -- 172.21.15.155:0/3603242749 shutdown_connections 2024-03-29T11:38:08.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.484+0000 7fc40e24c640 1 -- 172.21.15.155:0/3603242749 wait complete. 2024-03-29T11:38:08.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:08.485+0000 7fc40e24c640 1 librados: shutdown 2024-03-29T11:38:08.501 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:15.310 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:15.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.459+0000 7ff3ca484640 10 monclient: get_monmap_and_config 2024-03-29T11:38:15.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.459+0000 7ff3ca484640 10 monclient: build_initial_monmap 2024-03-29T11:38:15.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.459+0000 7ff3ca484640 10 monclient: monmap: 2024-03-29T11:38:15.460 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:15.460 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:15.460 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:38:15.460080+0000 2024-03-29T11:38:15.460 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:38:15.460080+0000 2024-03-29T11:38:15.460 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:15.460 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:15.460 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:15.460 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:38:15.460 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:38:15.460 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:15.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.462+0000 7ff3ca484640 1 Processor -- start 2024-03-29T11:38:15.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.462+0000 7ff3ca484640 1 -- start start 2024-03-29T11:38:15.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.462+0000 7ff3ca484640 10 monclient: init 2024-03-29T11:38:15.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.463+0000 7ff3ca484640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:15.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.463+0000 7ff3ca484640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:15.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.464+0000 7ff3ca484640 10 monclient(hunting): picked mon.noname-b con 0x7ff3c41457f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:15.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.464+0000 7ff3ca484640 10 monclient(hunting): picked mon.noname-c con 0x7ff3c413f650 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:15.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.464+0000 7ff3ca484640 10 monclient(hunting): picked mon.noname-a con 0x7ff3c40a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:15.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.464+0000 7ff3ca484640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c40aaef0 con 0x7ff3c40a9d70 2024-03-29T11:38:15.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.464+0000 7ff3ca484640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c4145bd0 con 0x7ff3c413f650 2024-03-29T11:38:15.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.464+0000 7ff3ca484640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c4145eb0 con 0x7ff3c41457f0 2024-03-29T11:38:15.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.464+0000 7ff3ca484640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:15.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.464+0000 7ff3ca484640 10 monclient(hunting): authenticate will time out at 1077.590739s 2024-03-29T11:38:15.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.466+0000 7ff3c3fff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ff3c41457f0 0x7ff3c413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50500/0 (socket says 172.21.15.155:50500) 2024-03-29T11:38:15.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.466+0000 7ff3c3fff640 1 -- 172.21.15.155:0/3257578670 learned_addr learned my addr 172.21.15.155:0/3257578670 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:15.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.467+0000 7ff3c2ffd640 1 -- 172.21.15.155:0/3257578670 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357185579 0 0) 0x7ff3c4145eb0 con 0x7ff3c41457f0 2024-03-29T11:38:15.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.467+0000 7ff3c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:15.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.467+0000 7ff3c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:15.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.467+0000 7ff3c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:15.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.467+0000 7ff3c2ffd640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:38:15.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.467+0000 7ff3c2ffd640 1 -- 172.21.15.155:0/3257578670 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ac003520 con 0x7ff3c41457f0 2024-03-29T11:38:15.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.467+0000 7ff3c2ffd640 1 -- 172.21.15.155:0/3257578670 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745117736 0 0) 0x7ff3ac003520 con 0x7ff3c41457f0 2024-03-29T11:38:15.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.467+0000 7ff3c2ffd640 1 -- 172.21.15.155:0/3257578670 >> v1:172.21.15.183:6789/0 conn(0x7ff3c413f650 legacy=0x7ff3c40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:15.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.467+0000 7ff3c2ffd640 1 -- 172.21.15.155:0/3257578670 >> v1:172.21.15.155:6789/0 conn(0x7ff3c40a9d70 legacy=0x7ff3c414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:15.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:15.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:38:15.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:15.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:38:15.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 1 -- 172.21.15.155:0/3257578670 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c4146340 con 0x7ff3c41457f0 2024-03-29T11:38:15.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:15.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:15.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:15.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3ca484640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:38:15.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3ca484640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 1 -- 172.21.15.155:0/3257578670 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff3b8001690 con 0x7ff3c41457f0 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.468+0000 7ff3c2ffd640 10 monclient: dump: 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:15.471 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:15.472 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:15.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.469+0000 7ff3ca484640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:15.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.469+0000 7ff3c2ffd640 1 -- 172.21.15.155:0/3257578670 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3b8001920 con 0x7ff3c41457f0 2024-03-29T11:38:15.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.469+0000 7ff3c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:15.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.469+0000 7ff3ca484640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:15.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.469+0000 7ff3ca484640 4 set_mon_vals no callback set 2024-03-29T11:38:15.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.470+0000 7ff3ca484640 10 monclient: shutdown 2024-03-29T11:38:15.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.470+0000 7ff3ca484640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:15.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.470+0000 7ff3ca484640 1 -- 172.21.15.155:0/3257578670 >> v1:172.21.15.155:6790/0 conn(0x7ff3c41457f0 legacy=0x7ff3c413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:15.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.470+0000 7ff3ca484640 1 -- 172.21.15.155:0/3257578670 shutdown_connections 2024-03-29T11:38:15.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.470+0000 7ff3ca484640 1 -- 172.21.15.155:0/3257578670 >> 172.21.15.155:0/3257578670 conn(0x7ff3c4066a00 msgr2=0x7ff3c4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:15.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.471+0000 7ff3ca484640 1 -- 172.21.15.155:0/3257578670 shutdown_connections 2024-03-29T11:38:15.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.471+0000 7ff3ca484640 1 -- 172.21.15.155:0/3257578670 wait complete. 2024-03-29T11:38:15.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.472+0000 7ff3ca484640 10 monclient: build_initial_monmap 2024-03-29T11:38:15.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.472+0000 7ff3ca484640 10 monclient: monmap: 2024-03-29T11:38:15.473 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.472+0000 7ff3ca484640 1 librados: starting msgr at 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.472+0000 7ff3ca484640 1 librados: starting objecter 2024-03-29T11:38:15.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.473+0000 7ff3ca484640 1 Processor -- start 2024-03-29T11:38:15.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.475+0000 7ff3ca484640 1 -- start start 2024-03-29T11:38:15.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.475+0000 7ff3ca484640 1 librados: setting wanted keys 2024-03-29T11:38:15.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.475+0000 7ff3ca484640 1 librados: calling monclient init 2024-03-29T11:38:15.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.475+0000 7ff3ca484640 10 monclient: init 2024-03-29T11:38:15.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.476+0000 7ff3ca484640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:15.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.476+0000 7ff3ca484640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:15.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.476+0000 7ff3ca484640 10 monclient(hunting): picked mon.noname-b con 0x7ff3c40a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:15.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.476+0000 7ff3ca484640 10 monclient(hunting): picked mon.noname-c con 0x7ff3c41457f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:15.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.476+0000 7ff3ca484640 10 monclient(hunting): picked mon.noname-a con 0x7ff3c4071640 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:15.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.476+0000 7ff3ca484640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c40abfc0 con 0x7ff3c4071640 2024-03-29T11:38:15.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.476+0000 7ff3ca484640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c41ded80 con 0x7ff3c40a9d70 2024-03-29T11:38:15.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.476+0000 7ff3ca484640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c4071a90 con 0x7ff3c41457f0 2024-03-29T11:38:15.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.476+0000 7ff3ca484640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:15.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.476+0000 7ff3ca484640 10 monclient(hunting): authenticate will time out at 1077.602427s 2024-03-29T11:38:15.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c37fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ff3c41457f0 0x7ff3c4070f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51178/0 (socket says 172.21.15.155:51178) 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c37fe640 1 -- 172.21.15.155:0/21060756 learned_addr learned my addr 172.21.15.155:0/21060756 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c0ff9640 10 client.?.objecter ms_handle_connect 0x7ff3c41457f0 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566972998 0 0) 0x7ff3c4071a90 con 0x7ff3c41457f0 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c0ff9640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b0004eb0 con 0x7ff3c41457f0 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c0ff9640 10 client.?.objecter ms_handle_connect 0x7ff3c40a9d70 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.479+0000 7ff3c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 client.?.objecter ms_handle_connect 0x7ff3c4071640 2024-03-29T11:38:15.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140905311 0 0) 0x7ff3c41ded80 con 0x7ff3c40a9d70 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c4071a90 con 0x7ff3c40a9d70 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2015716261 0 0) 0x7ff3b0004eb0 con 0x7ff3c41457f0 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3c41ded80 con 0x7ff3c41457f0 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff3bc001c20 con 0x7ff3c41457f0 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:15.482 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928833291 0 0) 0x7ff3c40abfc0 con 0x7ff3c4071640 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b0004eb0 con 0x7ff3c4071640 2024-03-29T11:38:15.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 178019292 0 0) 0x7ff3c41ded80 con 0x7ff3c41457f0 2024-03-29T11:38:15.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 >> v1:172.21.15.183:6789/0 conn(0x7ff3c40a9d70 legacy=0x7ff3c41df2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:15.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 >> v1:172.21.15.155:6789/0 conn(0x7ff3c4071640 legacy=0x7ff3c41e61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:15.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:15.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 monclient: found mon.c 2024-03-29T11:38:15.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:15.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:38:15.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c406de90 con 0x7ff3c41457f0 2024-03-29T11:38:15.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:15.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:15.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.480+0000 7ff3ca484640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.481+0000 7ff3ca484640 10 monclient: _renew_subs 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.481+0000 7ff3ca484640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.481+0000 7ff3ca484640 1 -- 172.21.15.155:0/21060756 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c41df9b0 con 0x7ff3c41457f0 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.481+0000 7ff3ca484640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.481+0000 7ff3ca484640 10 monclient: _renew_subs 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.481+0000 7ff3ca484640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.481+0000 7ff3ca484640 1 -- 172.21.15.155:0/21060756 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c41e7af0 con 0x7ff3c41457f0 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.481+0000 7ff3ca484640 1 librados: init done 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.483+0000 7ff3ca484640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.483+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3bc003970 con 0x7ff3c41457f0 2024-03-29T11:38:15.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.483+0000 7ff3c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:15.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.483+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff3bc003d80 con 0x7ff3c41457f0 2024-03-29T11:38:15.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.483+0000 7ff3c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:15.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.483+0000 7ff3c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:15.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.483+0000 7ff3c0ff9640 10 monclient: dump: 2024-03-29T11:38:15.487 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:15.487 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:15.487 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:15.487 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:15.488 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:15.488 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:15.488 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:15.488 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:15.488 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:15.488 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:15.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.483+0000 7ff3c27fc640 4 set_mon_vals no callback set 2024-03-29T11:38:15.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.484+0000 7ff3ca484640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:15.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.484+0000 7ff3ca484640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:38:15.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.484+0000 7ff3ca484640 1 -- 172.21.15.155:0/21060756 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c4075f10 con 0x7ff3c41457f0 2024-03-29T11:38:15.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.487+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7ff3bc003ee0 con 0x7ff3c41457f0 2024-03-29T11:38:15.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.488+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7ff3bc084d30 con 0x7ff3c41457f0 2024-03-29T11:38:15.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.488+0000 7ff3c0ff9640 10 client.4367.objecter ms_dispatch 0x7ff3c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:15.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.488+0000 7ff3c0ff9640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:15.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.488+0000 7ff3c0ff9640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:15.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.488+0000 7ff3c0ff9640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-29T11:38:15.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.491+0000 7ff3c3fff640 10 monclient: get_auth_request con 0x7ff3b006ded0 auth_method 0 2024-03-29T11:38:15.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.491+0000 7ff3c0ff9640 10 client.4367.objecter ms_handle_connect 0x7ff3b006ded0 2024-03-29T11:38:15.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.495+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3bc050610 con 0x7ff3c41457f0 2024-03-29T11:38:15.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.495+0000 7ff3c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:15.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.495+0000 7ff3c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:15.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.831+0000 7ff3ca484640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:15.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.831+0000 7ff3ca484640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:15.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.831+0000 7ff3ca484640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:38:15.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.831+0000 7ff3ca484640 1 -- 172.21.15.155:0/21060756 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3c41dfb40 con 0x7ff3c41457f0 2024-03-29T11:38:15.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.833+0000 7ff3c0ff9640 1 -- 172.21.15.155:0/21060756 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7ff3bc052540 con 0x7ff3c41457f0 2024-03-29T11:38:15.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.833+0000 7ff3c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:15.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.833+0000 7ff3c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:15.833 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:38:15.834 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:15.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.837+0000 7ff3ca484640 10 librados: watch_flush enter 2024-03-29T11:38:15.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.837+0000 7ff3ca484640 10 librados: watch_flush exit 2024-03-29T11:38:15.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.837+0000 7ff3ca484640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-29T11:38:15.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.837+0000 7ff3ca484640 10 client.4367.objecter successfully canceled tick 2024-03-29T11:38:15.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.837+0000 7ff3ca484640 1 -- 172.21.15.155:0/21060756 >> v1:172.21.15.155:6816/1760814708 conn(0x7ff3b006ded0 legacy=0x7ff3b0070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:15.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.837+0000 7ff3ca484640 10 monclient: shutdown 2024-03-29T11:38:15.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.837+0000 7ff3ca484640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:15.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.837+0000 7ff3ca484640 1 -- 172.21.15.155:0/21060756 >> v1:172.21.15.155:6790/0 conn(0x7ff3c41457f0 legacy=0x7ff3c4070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:15.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.838+0000 7ff3ca484640 1 -- 172.21.15.155:0/21060756 shutdown_connections 2024-03-29T11:38:15.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.838+0000 7ff3ca484640 1 -- 172.21.15.155:0/21060756 >> 172.21.15.155:0/21060756 conn(0x7ff3c4066a00 msgr2=0x7ff3c40b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:15.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.838+0000 7ff3ca484640 1 -- 172.21.15.155:0/21060756 shutdown_connections 2024-03-29T11:38:15.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.838+0000 7ff3ca484640 1 -- 172.21.15.155:0/21060756 wait complete. 2024-03-29T11:38:15.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:15.838+0000 7ff3ca484640 1 librados: shutdown 2024-03-29T11:38:15.858 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:22.667 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:22.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.818+0000 7f1e67fff640 10 monclient: get_monmap_and_config 2024-03-29T11:38:22.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.818+0000 7f1e67fff640 10 monclient: build_initial_monmap 2024-03-29T11:38:22.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.818+0000 7f1e67fff640 10 monclient: monmap: 2024-03-29T11:38:22.819 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:22.819 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:22.820 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:38:22.819873+0000 2024-03-29T11:38:22.820 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:38:22.819873+0000 2024-03-29T11:38:22.820 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:22.820 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:22.820 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:22.820 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:38:22.820 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:38:22.820 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:22.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.819+0000 7f1e67fff640 1 Processor -- start 2024-03-29T11:38:22.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.819+0000 7f1e67fff640 1 -- start start 2024-03-29T11:38:22.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.819+0000 7f1e67fff640 10 monclient: init 2024-03-29T11:38:22.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.820+0000 7f1e67fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:22.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.820+0000 7f1e67fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:22.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.820+0000 7f1e67fff640 10 monclient(hunting): picked mon.noname-c con 0x7f1e60006e30 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:22.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.820+0000 7f1e67fff640 10 monclient(hunting): picked mon.noname-b con 0x7f1e60035e00 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:22.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.820+0000 7f1e67fff640 10 monclient(hunting): picked mon.noname-a con 0x7f1e600c9670 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:22.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.820+0000 7f1e67fff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e60035970 con 0x7f1e600c9670 2024-03-29T11:38:22.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.820+0000 7f1e67fff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e60036280 con 0x7f1e60006e30 2024-03-29T11:38:22.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.820+0000 7f1e67fff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e60035640 con 0x7f1e60035e00 2024-03-29T11:38:22.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.820+0000 7f1e67fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:22.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.820+0000 7f1e67fff640 10 monclient(hunting): authenticate will time out at 1084.947444s 2024-03-29T11:38:22.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.821+0000 7f1e667fc640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f1e60035e00 0x7f1e600c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36440/0 (socket says 172.21.15.155:36440) 2024-03-29T11:38:22.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.821+0000 7f1e667fc640 1 -- 172.21.15.155:0/2271346166 learned_addr learned my addr 172.21.15.155:0/2271346166 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:22.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 1 -- 172.21.15.155:0/2271346166 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029102086 0 0) 0x7f1e60035640 con 0x7f1e60035e00 2024-03-29T11:38:22.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:22.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:22.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:22.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:38:22.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 1 -- 172.21.15.155:0/2271346166 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e50003520 con 0x7f1e60035e00 2024-03-29T11:38:22.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 1 -- 172.21.15.155:0/2271346166 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553474417 0 0) 0x7f1e60035970 con 0x7f1e600c9670 2024-03-29T11:38:22.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:22.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:22.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:22.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:38:22.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.825+0000 7f1e65ffb640 1 -- 172.21.15.155:0/2271346166 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e60035640 con 0x7f1e600c9670 2024-03-29T11:38:22.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 1 -- 172.21.15.155:0/2271346166 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520647108 0 0) 0x7f1e50003520 con 0x7f1e60035e00 2024-03-29T11:38:22.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 1 -- 172.21.15.155:0/2271346166 >> v1:172.21.15.183:6789/0 conn(0x7f1e60006e30 legacy=0x7f1e600cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 1 -- 172.21.15.155:0/2271346166 >> v1:172.21.15.155:6789/0 conn(0x7f1e600c9670 legacy=0x7f1e600d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 1 monclient: found mon.noname-b 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 1 -- 172.21.15.155:0/2271346166 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e60034e20 con 0x7f1e60035e00 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e67fff640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e67fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 1 -- 172.21.15.155:0/2271346166 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1e5c001610 con 0x7f1e60035e00 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 10 monclient: dump: 2024-03-29T11:38:22.828 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e67fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 1 -- 172.21.15.155:0/2271346166 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e5c0017e0 con 0x7f1e60035e00 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e67fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.826+0000 7f1e67fff640 4 set_mon_vals no callback set 2024-03-29T11:38:22.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.827+0000 7f1e65ffb640 1 -- 172.21.15.155:0/2271346166 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1e5c001bf0 con 0x7f1e60035e00 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.827+0000 7f1e65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.827+0000 7f1e65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.827+0000 7f1e65ffb640 10 monclient: dump: 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:22.830 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:22.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.827+0000 7f1e67fff640 10 monclient: shutdown 2024-03-29T11:38:22.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.827+0000 7f1e67fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:22.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.827+0000 7f1e67fff640 1 -- 172.21.15.155:0/2271346166 >> v1:172.21.15.155:6790/0 conn(0x7f1e60035e00 legacy=0x7f1e600c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.827+0000 7f1e67fff640 1 -- 172.21.15.155:0/2271346166 shutdown_connections 2024-03-29T11:38:22.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.827+0000 7f1e67fff640 1 -- 172.21.15.155:0/2271346166 >> 172.21.15.155:0/2271346166 conn(0x7f1e60005c00 msgr2=0x7f1e60006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:22.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.828+0000 7f1e67fff640 1 -- 172.21.15.155:0/2271346166 shutdown_connections 2024-03-29T11:38:22.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.828+0000 7f1e67fff640 1 -- 172.21.15.155:0/2271346166 wait complete. 2024-03-29T11:38:22.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.829+0000 7f1e67fff640 10 monclient: build_initial_monmap 2024-03-29T11:38:22.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.829+0000 7f1e67fff640 10 monclient: monmap: 2024-03-29T11:38:22.832 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.830+0000 7f1e67fff640 1 librados: starting msgr at 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.830+0000 7f1e67fff640 1 librados: starting objecter 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.830+0000 7f1e67fff640 1 Processor -- start 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.830+0000 7f1e67fff640 1 -- start start 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.830+0000 7f1e67fff640 1 librados: setting wanted keys 2024-03-29T11:38:22.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.830+0000 7f1e67fff640 1 librados: calling monclient init 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.830+0000 7f1e67fff640 10 monclient: init 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e67fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e67fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e67fff640 10 monclient(hunting): picked mon.noname-b con 0x7f1e60006e30 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e67fff640 10 monclient(hunting): picked mon.noname-c con 0x7f1e60035e00 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e67fff640 10 monclient(hunting): picked mon.noname-a con 0x7f1e600c9670 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e67fff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e600c0620 con 0x7f1e600c9670 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e67fff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6002e6d0 con 0x7f1e60006e30 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e67fff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6002e190 con 0x7f1e60035e00 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e67fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e67fff640 10 monclient(hunting): authenticate will time out at 1084.958227s 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e667fc640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f1e60035e00 0x7f1e6002eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:38018/0 (socket says 172.21.15.155:38018) 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e667fc640 1 -- 172.21.15.155:0/3193733975 learned_addr learned my addr 172.21.15.155:0/3193733975 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:22.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e4f7fe640 10 client.?.objecter ms_handle_connect 0x7f1e60035e00 2024-03-29T11:38:22.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:22.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233596967 0 0) 0x7f1e6002e190 con 0x7f1e60035e00 2024-03-29T11:38:22.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e4f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:22.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:22.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:22.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e4f7fe640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:38:22.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.831+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e54004eb0 con 0x7f1e60035e00 2024-03-29T11:38:22.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.833+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3714665967 0 0) 0x7f1e54004eb0 con 0x7f1e60035e00 2024-03-29T11:38:22.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.834+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e6002e190 con 0x7f1e60035e00 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.834+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1e5c001af0 con 0x7f1e60035e00 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.834+0000 7f1e4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.834+0000 7f1e4f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.834+0000 7f1e4f7fe640 10 monclient(hunting): dump: 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:22.836 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:22.837 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:22.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.834+0000 7f1e4f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:22.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 344686576 0 0) 0x7f1e6002e190 con 0x7f1e60035e00 2024-03-29T11:38:22.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 >> v1:172.21.15.183:6789/0 conn(0x7f1e60006e30 legacy=0x7f1e60033670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:38:22.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 >> v1:172.21.15.155:6789/0 conn(0x7f1e600c9670 legacy=0x7f1e60185a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:22.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 1 monclient: found mon.c 2024-03-29T11:38:22.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:22.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:38:22.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e600c3db0 con 0x7f1e60035e00 2024-03-29T11:38:22.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:22.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:22.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:22.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e5c004a90 con 0x7f1e60035e00 2024-03-29T11:38:22.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:22.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1e5c004ee0 con 0x7f1e60035e00 2024-03-29T11:38:22.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:22.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e4f7fe640 10 monclient: dump: 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:22.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.837+0000 7f1e657fa640 4 set_mon_vals no callback set 2024-03-29T11:38:22.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.839+0000 7f1e67fff640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:38:22.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.839+0000 7f1e67fff640 10 monclient: _renew_subs 2024-03-29T11:38:22.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.839+0000 7f1e67fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:38:22.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.839+0000 7f1e67fff640 1 -- 172.21.15.155:0/3193733975 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e6002f270 con 0x7f1e60035e00 2024-03-29T11:38:22.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.839+0000 7f1e67fff640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:22.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.839+0000 7f1e67fff640 10 monclient: _renew_subs 2024-03-29T11:38:22.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.839+0000 7f1e67fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:38:22.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.839+0000 7f1e67fff640 1 -- 172.21.15.155:0/3193733975 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e600c52d0 con 0x7f1e60035e00 2024-03-29T11:38:22.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.839+0000 7f1e67fff640 1 librados: init done 2024-03-29T11:38:22.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.841+0000 7f1e67fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:22.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.841+0000 7f1e67fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:22.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.841+0000 7f1e67fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:38:22.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.841+0000 7f1e67fff640 1 -- 172.21.15.155:0/3193733975 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e60007b90 con 0x7f1e60035e00 2024-03-29T11:38:22.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.842+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f1e5c005040 con 0x7f1e60035e00 2024-03-29T11:38:22.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.843+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f1e5c084be0 con 0x7f1e60035e00 2024-03-29T11:38:22.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.843+0000 7f1e4f7fe640 10 client.4373.objecter ms_dispatch 0x7f1e68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:22.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.843+0000 7f1e4f7fe640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:22.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.843+0000 7f1e4f7fe640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:22.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.843+0000 7f1e4f7fe640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T11:38:22.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.843+0000 7f1e66ffd640 10 monclient: get_auth_request con 0x7f1e5406daf0 auth_method 0 2024-03-29T11:38:22.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.843+0000 7f1e4f7fe640 10 client.4373.objecter ms_handle_connect 0x7f1e5406daf0 2024-03-29T11:38:22.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.854+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e5c0504c0 con 0x7f1e60035e00 2024-03-29T11:38:22.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.854+0000 7f1e4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:22.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:22.854+0000 7f1e4f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:23.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.177+0000 7f1e67fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:23.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.177+0000 7f1e67fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:23.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.177+0000 7f1e67fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:38:23.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.177+0000 7f1e67fff640 1 -- 172.21.15.155:0/3193733975 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e6003ac20 con 0x7f1e60035e00 2024-03-29T11:38:23.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.179+0000 7f1e4f7fe640 1 -- 172.21.15.155:0/3193733975 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f1e5c0523f0 con 0x7f1e60035e00 2024-03-29T11:38:23.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.179+0000 7f1e4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:23.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.179+0000 7f1e4f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:23.183 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:38:23.183 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:23.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.183+0000 7f1e67fff640 10 librados: watch_flush enter 2024-03-29T11:38:23.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.183+0000 7f1e67fff640 10 librados: watch_flush exit 2024-03-29T11:38:23.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.183+0000 7f1e67fff640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T11:38:23.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.183+0000 7f1e67fff640 10 client.4373.objecter successfully canceled tick 2024-03-29T11:38:23.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.183+0000 7f1e67fff640 1 -- 172.21.15.155:0/3193733975 >> v1:172.21.15.155:6816/1760814708 conn(0x7f1e5406daf0 legacy=0x7f1e5406ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:23.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.183+0000 7f1e67fff640 10 monclient: shutdown 2024-03-29T11:38:23.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.183+0000 7f1e67fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:23.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.183+0000 7f1e67fff640 1 -- 172.21.15.155:0/3193733975 >> v1:172.21.15.155:6790/0 conn(0x7f1e60035e00 legacy=0x7f1e6002eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:23.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.184+0000 7f1e67fff640 1 -- 172.21.15.155:0/3193733975 shutdown_connections 2024-03-29T11:38:23.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.184+0000 7f1e67fff640 1 -- 172.21.15.155:0/3193733975 >> 172.21.15.155:0/3193733975 conn(0x7f1e60005c00 msgr2=0x7f1e600c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:23.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.184+0000 7f1e67fff640 1 -- 172.21.15.155:0/3193733975 shutdown_connections 2024-03-29T11:38:23.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.184+0000 7f1e67fff640 1 -- 172.21.15.155:0/3193733975 wait complete. 2024-03-29T11:38:23.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:23.184+0000 7f1e67fff640 1 librados: shutdown 2024-03-29T11:38:23.202 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:30.009 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:30.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.167+0000 7f076bb59640 10 monclient: get_monmap_and_config 2024-03-29T11:38:30.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.167+0000 7f076bb59640 10 monclient: build_initial_monmap 2024-03-29T11:38:30.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.167+0000 7f076bb59640 10 monclient: monmap: 2024-03-29T11:38:30.168 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:30.169 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:30.169 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:38:30.168644+0000 2024-03-29T11:38:30.169 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:38:30.168644+0000 2024-03-29T11:38:30.169 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:30.169 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:30.169 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:30.169 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:38:30.169 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:38:30.169 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:30.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.168+0000 7f076bb59640 1 Processor -- start 2024-03-29T11:38:30.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.168+0000 7f076bb59640 1 -- start start 2024-03-29T11:38:30.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.168+0000 7f076bb59640 10 monclient: init 2024-03-29T11:38:30.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.169+0000 7f076bb59640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:30.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.169+0000 7f076bb59640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:30.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.169+0000 7f076bb59640 10 monclient(hunting): picked mon.noname-a con 0x7f07640b0860 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:30.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.169+0000 7f076bb59640 10 monclient(hunting): picked mon.noname-c con 0x7f076413ef30 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:30.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.169+0000 7f076bb59640 10 monclient(hunting): picked mon.noname-b con 0x7f07640af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:30.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.169+0000 7f076bb59640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07640b0c40 con 0x7f07640b0860 2024-03-29T11:38:30.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.169+0000 7f076bb59640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07640aed10 con 0x7f076413ef30 2024-03-29T11:38:30.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.169+0000 7f076bb59640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07640afd50 con 0x7f07640af8e0 2024-03-29T11:38:30.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.169+0000 7f076bb59640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:30.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.169+0000 7f076bb59640 10 monclient(hunting): authenticate will time out at 1092.296252s 2024-03-29T11:38:30.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.173+0000 7f07690f4640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f076413ef30 0x7f07640af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:45489/0 (socket says 172.21.15.155:45489) 2024-03-29T11:38:30.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.173+0000 7f07690f4640 1 -- 172.21.15.155:0/2717459354 learned_addr learned my addr 172.21.15.155:0/2717459354 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:30.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.176+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645583875 0 0) 0x7f07640aed10 con 0x7f076413ef30 2024-03-29T11:38:30.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.176+0000 7f07688f3640 10 monclient(hunting): _negotiate 2024-03-29T11:38:30.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.176+0000 7f07688f3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:30.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.176+0000 7f07688f3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:30.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.176+0000 7f07688f3640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:38:30.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.176+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0750003520 con 0x7f076413ef30 2024-03-29T11:38:30.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259489500 0 0) 0x7f07640b0c40 con 0x7f07640b0860 2024-03-29T11:38:30.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient(hunting): _negotiate 2024-03-29T11:38:30.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:30.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:30.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:38:30.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07640aed10 con 0x7f07640b0860 2024-03-29T11:38:30.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504074977 0 0) 0x7f07640afd50 con 0x7f07640af8e0 2024-03-29T11:38:30.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient(hunting): _negotiate 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07640b0c40 con 0x7f07640af8e0 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903495432 0 0) 0x7f07640aed10 con 0x7f07640b0860 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 >> v1:172.21.15.155:6790/0 conn(0x7f07640af8e0 legacy=0x7f0764149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 >> v1:172.21.15.183:6789/0 conn(0x7f076413ef30 legacy=0x7f07640af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient: _finish_hunting 0 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 monclient: found mon.noname-a 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:38:30.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07640ab1b0 con 0x7f07640b0860 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient: _finish_auth 0 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient: _check_auth_tickets 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f076bb59640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f076bb59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0760001670 con 0x7f07640b0860 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient: dump: 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:30.184 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f076bb59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07600017d0 con 0x7f07640b0860 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f076bb59640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 1 -- 172.21.15.155:0/2717459354 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0760001be0 con 0x7f07640b0860 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:30.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f076bb59640 4 set_mon_vals no callback set 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.177+0000 7f07688f3640 10 monclient: dump: 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:30.186 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:30.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.178+0000 7f076bb59640 10 monclient: shutdown 2024-03-29T11:38:30.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.178+0000 7f076bb59640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:30.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.178+0000 7f076bb59640 1 -- 172.21.15.155:0/2717459354 >> v1:172.21.15.155:6789/0 conn(0x7f07640b0860 legacy=0x7f076413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:30.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.178+0000 7f076bb59640 1 -- 172.21.15.155:0/2717459354 shutdown_connections 2024-03-29T11:38:30.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.178+0000 7f076bb59640 1 -- 172.21.15.155:0/2717459354 >> 172.21.15.155:0/2717459354 conn(0x7f076406b660 msgr2=0x7f076406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:30.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.179+0000 7f076bb59640 1 -- 172.21.15.155:0/2717459354 shutdown_connections 2024-03-29T11:38:30.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.179+0000 7f076bb59640 1 -- 172.21.15.155:0/2717459354 wait complete. 2024-03-29T11:38:30.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.180+0000 7f076bb59640 10 monclient: build_initial_monmap 2024-03-29T11:38:30.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.180+0000 7f076bb59640 10 monclient: monmap: 2024-03-29T11:38:30.190 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:30.190 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:30.190 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:38:30.190 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:38:30.190 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:30.190 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:30.190 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:30.190 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.180+0000 7f076bb59640 1 librados: starting msgr at 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.180+0000 7f076bb59640 1 librados: starting objecter 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.181+0000 7f076bb59640 1 Processor -- start 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.181+0000 7f076bb59640 1 -- start start 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.181+0000 7f076bb59640 1 librados: setting wanted keys 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.181+0000 7f076bb59640 1 librados: calling monclient init 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.181+0000 7f076bb59640 10 monclient: init 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.181+0000 7f076bb59640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.181+0000 7f076bb59640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.181+0000 7f076bb59640 10 monclient(hunting): picked mon.noname-c con 0x7f07640af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:30.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.181+0000 7f076bb59640 10 monclient(hunting): picked mon.noname-a con 0x7f07640b0860 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.181+0000 7f076bb59640 10 monclient(hunting): picked mon.noname-b con 0x7f076413ef30 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.182+0000 7f076bb59640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07640b35c0 con 0x7f07640b0860 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.182+0000 7f076bb59640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07641c4a50 con 0x7f076413ef30 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.182+0000 7f076bb59640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07641e17b0 con 0x7f07640af8e0 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.182+0000 7f076bb59640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.182+0000 7f076bb59640 10 monclient(hunting): authenticate will time out at 1092.308894s 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.188+0000 7f07690f4640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f07640b0860 0x7f07641e7260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:40793/0 (socket says 172.21.15.155:40793) 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.188+0000 7f07690f4640 1 -- 172.21.15.155:0/375860398 learned_addr learned my addr 172.21.15.155:0/375860398 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.189+0000 7f075a7fc640 10 client.?.objecter ms_handle_connect 0x7f076413ef30 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.189+0000 7f075a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.189+0000 7f075a7fc640 10 client.?.objecter ms_handle_connect 0x7f07640b0860 2024-03-29T11:38:30.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.189+0000 7f075a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:30.194 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.190+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585198416 0 0) 0x7f07641c4a50 con 0x7f076413ef30 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.190+0000 7f075a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.190+0000 7f075a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.190+0000 7f075a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.190+0000 7f075a7fc640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.190+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0748004eb0 con 0x7f076413ef30 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.191+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1485080063 0 0) 0x7f0748004eb0 con 0x7f076413ef30 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.191+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07641c4a50 con 0x7f076413ef30 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.191+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0754001a10 con 0x7f076413ef30 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.191+0000 7f075a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.191+0000 7f075a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.191+0000 7f075a7fc640 10 monclient(hunting): dump: 2024-03-29T11:38:30.195 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.191+0000 7f075a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.191+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2023701670 0 0) 0x7f07641c4a50 con 0x7f076413ef30 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.191+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 >> v1:172.21.15.155:6790/0 conn(0x7f07640af8e0 legacy=0x7f07641e10a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 >> v1:172.21.15.155:6789/0 conn(0x7f07640b0860 legacy=0x7f07641e7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:30.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 1 monclient: found mon.b 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07641e1940 con 0x7f076413ef30 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07540035f0 con 0x7f076413ef30 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0754003900 con 0x7f076413ef30 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.192+0000 7f075a7fc640 10 monclient: dump: 2024-03-29T11:38:30.197 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:30.198 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:30.198 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:30.198 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:30.198 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:30.198 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:30.198 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:30.198 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:30.198 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:30.198 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:30.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.194+0000 7f076bb59640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:38:30.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.194+0000 7f076bb59640 10 monclient: _renew_subs 2024-03-29T11:38:30.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.194+0000 7f076bb59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:30.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.194+0000 7f076bb59640 1 -- 172.21.15.155:0/375860398 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07640b3010 con 0x7f076413ef30 2024-03-29T11:38:30.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.194+0000 7f076bb59640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:30.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.194+0000 7f076bb59640 10 monclient: _renew_subs 2024-03-29T11:38:30.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.194+0000 7f076bb59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:30.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.194+0000 7f076bb59640 1 -- 172.21.15.155:0/375860398 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07641373c0 con 0x7f076413ef30 2024-03-29T11:38:30.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.194+0000 7f076bb59640 1 librados: init done 2024-03-29T11:38:30.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.194+0000 7f075b7fe640 4 set_mon_vals no callback set 2024-03-29T11:38:30.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.196+0000 7f076bb59640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:30.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.196+0000 7f076bb59640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:30.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.196+0000 7f076bb59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:30.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.196+0000 7f076bb59640 1 -- 172.21.15.155:0/375860398 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0764075f10 con 0x7f076413ef30 2024-03-29T11:38:30.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.198+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f0754019950 con 0x7f076413ef30 2024-03-29T11:38:30.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.199+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f07540838e0 con 0x7f076413ef30 2024-03-29T11:38:30.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.199+0000 7f075a7fc640 10 client.4366.objecter ms_dispatch 0x7f0764000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:30.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.199+0000 7f075a7fc640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:30.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.199+0000 7f075a7fc640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:30.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.199+0000 7f075a7fc640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-29T11:38:30.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.200+0000 7f07698f5640 10 monclient: get_auth_request con 0x7f074806db30 auth_method 0 2024-03-29T11:38:30.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.200+0000 7f075a7fc640 10 client.4366.objecter ms_handle_connect 0x7f074806db30 2024-03-29T11:38:30.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.204+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f075404eff0 con 0x7f076413ef30 2024-03-29T11:38:30.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.204+0000 7f075a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:30.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.205+0000 7f075a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:30.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.533+0000 7f076bb59640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:30.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.534+0000 7f076bb59640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:30.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.534+0000 7f076bb59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:30.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.534+0000 7f076bb59640 1 -- 172.21.15.155:0/375860398 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0764072330 con 0x7f076413ef30 2024-03-29T11:38:30.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.535+0000 7f075a7fc640 1 -- 172.21.15.155:0/375860398 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f07540510f0 con 0x7f076413ef30 2024-03-29T11:38:30.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.535+0000 7f075a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:30.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.535+0000 7f075a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:30.538 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:38:30.538 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:30.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.540+0000 7f076bb59640 10 librados: watch_flush enter 2024-03-29T11:38:30.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.540+0000 7f076bb59640 10 librados: watch_flush exit 2024-03-29T11:38:30.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.540+0000 7f076bb59640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-29T11:38:30.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.540+0000 7f076bb59640 10 client.4366.objecter successfully canceled tick 2024-03-29T11:38:30.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.541+0000 7f076bb59640 1 -- 172.21.15.155:0/375860398 >> v1:172.21.15.155:6816/1760814708 conn(0x7f074806db30 legacy=0x7f074806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:30.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.541+0000 7f076bb59640 10 monclient: shutdown 2024-03-29T11:38:30.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.541+0000 7f076bb59640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:30.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.541+0000 7f076bb59640 1 -- 172.21.15.155:0/375860398 >> v1:172.21.15.183:6789/0 conn(0x7f076413ef30 legacy=0x7f07641ef5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:30.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.542+0000 7f076bb59640 1 -- 172.21.15.155:0/375860398 shutdown_connections 2024-03-29T11:38:30.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.542+0000 7f076bb59640 1 -- 172.21.15.155:0/375860398 >> 172.21.15.155:0/375860398 conn(0x7f076406b660 msgr2=0x7f076413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:30.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.546+0000 7f076bb59640 1 -- 172.21.15.155:0/375860398 shutdown_connections 2024-03-29T11:38:30.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.547+0000 7f076bb59640 1 -- 172.21.15.155:0/375860398 wait complete. 2024-03-29T11:38:30.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:30.547+0000 7f076bb59640 1 librados: shutdown 2024-03-29T11:38:30.566 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:37.373 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:37.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.513+0000 7f6b78820640 10 monclient: get_monmap_and_config 2024-03-29T11:38:37.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.513+0000 7f6b78820640 10 monclient: build_initial_monmap 2024-03-29T11:38:37.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.513+0000 7f6b78820640 10 monclient: monmap: 2024-03-29T11:38:37.515 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:37.515 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:37.516 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:38:37.514478+0000 2024-03-29T11:38:37.516 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:38:37.514478+0000 2024-03-29T11:38:37.516 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:37.516 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:37.516 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:37.516 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:38:37.516 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:38:37.516 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:37.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.514+0000 7f6b78820640 1 Processor -- start 2024-03-29T11:38:37.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.514+0000 7f6b78820640 1 -- start start 2024-03-29T11:38:37.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.514+0000 7f6b78820640 10 monclient: init 2024-03-29T11:38:37.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.516+0000 7f6b78820640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.516+0000 7f6b78820640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.516+0000 7f6b78820640 10 monclient(hunting): picked mon.noname-a con 0x7f6b7406d9d0 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.516+0000 7f6b78820640 10 monclient(hunting): picked mon.noname-b con 0x7f6b7413ef30 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.516+0000 7f6b78820640 10 monclient(hunting): picked mon.noname-c con 0x7f6b740b07d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.516+0000 7f6b78820640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b740aed10 con 0x7f6b7406d9d0 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.516+0000 7f6b78820640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b740b0c40 con 0x7f6b740b07d0 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.516+0000 7f6b78820640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b740ab1b0 con 0x7f6b7413ef30 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.516+0000 7f6b78820640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.516+0000 7f6b78820640 10 monclient(hunting): authenticate will time out at 1099.643241s 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.520+0000 7f6b71d9b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f6b7413ef30 0x7f6b740af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:54653/0 (socket says 172.21.15.155:54653) 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.520+0000 7f6b71d9b640 1 -- 172.21.15.155:0/3785879992 learned_addr learned my addr 172.21.15.155:0/3785879992 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.521+0000 7f6b7159a640 1 -- 172.21.15.155:0/3785879992 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954688873 0 0) 0x7f6b740ab1b0 con 0x7f6b7413ef30 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.521+0000 7f6b7159a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:37.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.521+0000 7f6b7159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:37.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.521+0000 7f6b7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:37.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.521+0000 7f6b7159a640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:38:37.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.521+0000 7f6b7159a640 1 -- 172.21.15.155:0/3785879992 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b5c003520 con 0x7f6b7413ef30 2024-03-29T11:38:37.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 1 -- 172.21.15.155:0/3785879992 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3611257000 0 0) 0x7f6b5c003520 con 0x7f6b7413ef30 2024-03-29T11:38:37.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 1 -- 172.21.15.155:0/3785879992 >> v1:172.21.15.183:6789/0 conn(0x7f6b740b07d0 legacy=0x7f6b7414bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 1 -- 172.21.15.155:0/3785879992 >> v1:172.21.15.155:6789/0 conn(0x7f6b7406d9d0 legacy=0x7f6b7413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:37.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 1 monclient: found mon.noname-b 2024-03-29T11:38:37.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:37.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:38:37.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 1 -- 172.21.15.155:0/3785879992 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b740ab490 con 0x7f6b7413ef30 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 10 monclient: _finish_auth 0 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 1 -- 172.21.15.155:0/3785879992 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6b68001690 con 0x7f6b7413ef30 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.522+0000 7f6b7159a640 10 monclient: dump: 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:37.525 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:37.526 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:37.526 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:37.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.524+0000 7f6b78820640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:38:37.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.524+0000 7f6b78820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:37.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.525+0000 7f6b7159a640 1 -- 172.21.15.155:0/3785879992 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b680017f0 con 0x7f6b7413ef30 2024-03-29T11:38:37.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.525+0000 7f6b7159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:37.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.525+0000 7f6b78820640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:37.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.525+0000 7f6b78820640 4 set_mon_vals no callback set 2024-03-29T11:38:37.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.525+0000 7f6b78820640 10 monclient: shutdown 2024-03-29T11:38:37.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.525+0000 7f6b78820640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:37.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.525+0000 7f6b78820640 1 -- 172.21.15.155:0/3785879992 >> v1:172.21.15.155:6790/0 conn(0x7f6b7413ef30 legacy=0x7f6b740af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.527+0000 7f6b78820640 1 -- 172.21.15.155:0/3785879992 shutdown_connections 2024-03-29T11:38:37.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.527+0000 7f6b78820640 1 -- 172.21.15.155:0/3785879992 >> 172.21.15.155:0/3785879992 conn(0x7f6b7406b660 msgr2=0x7f6b7406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:37.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.530+0000 7f6b78820640 1 -- 172.21.15.155:0/3785879992 shutdown_connections 2024-03-29T11:38:37.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.530+0000 7f6b78820640 1 -- 172.21.15.155:0/3785879992 wait complete. 2024-03-29T11:38:37.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.532+0000 7f6b78820640 10 monclient: build_initial_monmap 2024-03-29T11:38:37.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.532+0000 7f6b78820640 10 monclient: monmap: 2024-03-29T11:38:37.533 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:37.533 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:37.533 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:38:37.533 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:38:37.533 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:37.533 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:37.533 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:37.534 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:38:37.534 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:38:37.534 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:37.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.532+0000 7f6b78820640 1 librados: starting msgr at 2024-03-29T11:38:37.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.532+0000 7f6b78820640 1 librados: starting objecter 2024-03-29T11:38:37.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.532+0000 7f6b78820640 1 Processor -- start 2024-03-29T11:38:37.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.532+0000 7f6b78820640 1 -- start start 2024-03-29T11:38:37.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.532+0000 7f6b78820640 1 librados: setting wanted keys 2024-03-29T11:38:37.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.532+0000 7f6b78820640 1 librados: calling monclient init 2024-03-29T11:38:37.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.532+0000 7f6b78820640 10 monclient: init 2024-03-29T11:38:37.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.533+0000 7f6b78820640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:37.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.533+0000 7f6b78820640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:37.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.533+0000 7f6b78820640 10 monclient(hunting): picked mon.noname-a con 0x7f6b7406d9d0 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:37.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.533+0000 7f6b78820640 10 monclient(hunting): picked mon.noname-c con 0x7f6b740b07d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:37.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.533+0000 7f6b78820640 10 monclient(hunting): picked mon.noname-b con 0x7f6b7413ef30 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:37.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.533+0000 7f6b78820640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7406eaa0 con 0x7f6b7406d9d0 2024-03-29T11:38:37.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.533+0000 7f6b78820640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b74135200 con 0x7f6b7413ef30 2024-03-29T11:38:37.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.533+0000 7f6b78820640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7406fb70 con 0x7f6b740b07d0 2024-03-29T11:38:37.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.533+0000 7f6b78820640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:37.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.533+0000 7f6b78820640 10 monclient(hunting): authenticate will time out at 1099.660273s 2024-03-29T11:38:37.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.540+0000 7f6b7259c640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f6b7406d9d0 0x7f6b74070270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:33567/0 (socket says 172.21.15.155:33567) 2024-03-29T11:38:37.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.540+0000 7f6b7259c640 1 -- 172.21.15.155:0/1993787164 learned_addr learned my addr 172.21.15.155:0/1993787164 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:37.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.541+0000 7f6b62ffd640 10 client.?.objecter ms_handle_connect 0x7f6b7406d9d0 2024-03-29T11:38:37.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.541+0000 7f6b62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:37.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.541+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614869931 0 0) 0x7f6b7406eaa0 con 0x7f6b7406d9d0 2024-03-29T11:38:37.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.541+0000 7f6b62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:37.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.541+0000 7f6b62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:37.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.541+0000 7f6b62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:37.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.541+0000 7f6b62ffd640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:38:37.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.541+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b54004eb0 con 0x7f6b7406d9d0 2024-03-29T11:38:37.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.541+0000 7f6b62ffd640 10 client.?.objecter ms_handle_connect 0x7f6b740b07d0 2024-03-29T11:38:37.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.541+0000 7f6b62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.542+0000 7f6b62ffd640 10 client.?.objecter ms_handle_connect 0x7f6b7413ef30 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.542+0000 7f6b62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.542+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 955210122 0 0) 0x7f6b54004eb0 con 0x7f6b7406d9d0 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.542+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b7406eaa0 con 0x7f6b7406d9d0 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.542+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6b64001990 con 0x7f6b7406d9d0 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.542+0000 7f6b62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.542+0000 7f6b62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.542+0000 7f6b62ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:37.545 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.542+0000 7f6b62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.542+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2325137279 0 0) 0x7f6b7406eaa0 con 0x7f6b7406d9d0 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 >> v1:172.21.15.155:6790/0 conn(0x7f6b740b07d0 legacy=0x7f6b741343e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 >> v1:172.21.15.183:6789/0 conn(0x7f6b7413ef30 legacy=0x7f6b74134af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 1 monclient: found mon.a 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7406ec80 con 0x7f6b7406d9d0 2024-03-29T11:38:37.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b64001af0 con 0x7f6b7406d9d0 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6b64003a60 con 0x7f6b7406d9d0 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.543+0000 7f6b62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.544+0000 7f6b62ffd640 10 monclient: dump: 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:37.547 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:37.548 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:37.548 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:37.548 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:37.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.544+0000 7f6b63fff640 4 set_mon_vals no callback set 2024-03-29T11:38:37.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.545+0000 7f6b78820640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:38:37.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.545+0000 7f6b78820640 10 monclient: _renew_subs 2024-03-29T11:38:37.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.545+0000 7f6b78820640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:38:37.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.545+0000 7f6b78820640 1 -- 172.21.15.155:0/1993787164 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b740b3fc0 con 0x7f6b7406d9d0 2024-03-29T11:38:37.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.545+0000 7f6b78820640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:37.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.545+0000 7f6b78820640 10 monclient: _renew_subs 2024-03-29T11:38:37.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.545+0000 7f6b78820640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:38:37.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.545+0000 7f6b78820640 1 -- 172.21.15.155:0/1993787164 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7406e200 con 0x7f6b7406d9d0 2024-03-29T11:38:37.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.545+0000 7f6b78820640 1 librados: init done 2024-03-29T11:38:37.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.547+0000 7f6b78820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:37.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.547+0000 7f6b78820640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:37.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.547+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f6b64003bc0 con 0x7f6b7406d9d0 2024-03-29T11:38:37.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.547+0000 7f6b78820640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:38:37.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.547+0000 7f6b78820640 1 -- 172.21.15.155:0/1993787164 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7406ee30 con 0x7f6b7406d9d0 2024-03-29T11:38:37.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.548+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f6b64084b50 con 0x7f6b7406d9d0 2024-03-29T11:38:37.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.548+0000 7f6b62ffd640 10 client.4380.objecter ms_dispatch 0x7f6b74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:37.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.548+0000 7f6b62ffd640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:37.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.548+0000 7f6b62ffd640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:37.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.548+0000 7f6b62ffd640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T11:38:37.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.548+0000 7f6b71d9b640 10 monclient: get_auth_request con 0x7f6b5406dce0 auth_method 0 2024-03-29T11:38:37.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.548+0000 7f6b62ffd640 10 client.4380.objecter ms_handle_connect 0x7f6b5406dce0 2024-03-29T11:38:37.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.559+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b64050430 con 0x7f6b7406d9d0 2024-03-29T11:38:37.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.559+0000 7f6b62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:37.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.559+0000 7f6b62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:37.890 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.888+0000 7f6b78820640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:37.890 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.888+0000 7f6b78820640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:37.890 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.888+0000 7f6b78820640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:38:37.890 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.888+0000 7f6b78820640 1 -- 172.21.15.155:0/1993787164 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b74075f10 con 0x7f6b7406d9d0 2024-03-29T11:38:37.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.893+0000 7f6b62ffd640 1 -- 172.21.15.155:0/1993787164 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f6b64052360 con 0x7f6b7406d9d0 2024-03-29T11:38:37.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.893+0000 7f6b62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:37.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.893+0000 7f6b62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:37.897 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:38:37.898 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:37.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.900+0000 7f6b78820640 10 librados: watch_flush enter 2024-03-29T11:38:37.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.900+0000 7f6b78820640 10 librados: watch_flush exit 2024-03-29T11:38:37.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.900+0000 7f6b78820640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T11:38:37.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.900+0000 7f6b78820640 10 client.4380.objecter successfully canceled tick 2024-03-29T11:38:37.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.900+0000 7f6b78820640 1 -- 172.21.15.155:0/1993787164 >> v1:172.21.15.155:6816/1760814708 conn(0x7f6b5406dce0 legacy=0x7f6b540701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.900+0000 7f6b78820640 10 monclient: shutdown 2024-03-29T11:38:37.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.900+0000 7f6b78820640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:37.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.900+0000 7f6b78820640 1 -- 172.21.15.155:0/1993787164 >> v1:172.21.15.155:6789/0 conn(0x7f6b7406d9d0 legacy=0x7f6b74070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.902+0000 7f6b78820640 1 -- 172.21.15.155:0/1993787164 shutdown_connections 2024-03-29T11:38:37.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.902+0000 7f6b78820640 1 -- 172.21.15.155:0/1993787164 >> 172.21.15.155:0/1993787164 conn(0x7f6b7406b660 msgr2=0x7f6b7413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:37.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.902+0000 7f6b78820640 1 -- 172.21.15.155:0/1993787164 shutdown_connections 2024-03-29T11:38:37.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.902+0000 7f6b78820640 1 -- 172.21.15.155:0/1993787164 wait complete. 2024-03-29T11:38:37.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:37.902+0000 7f6b78820640 1 librados: shutdown 2024-03-29T11:38:37.920 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:44.727 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:44.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.892+0000 7f93b43d6640 10 monclient: get_monmap_and_config 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.892+0000 7f93b43d6640 10 monclient: build_initial_monmap 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.892+0000 7f93b43d6640 10 monclient: monmap: 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:38:44.894261+0000 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:38:44.894261+0000 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:38:44.894 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:44.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.893+0000 7f93b43d6640 1 Processor -- start 2024-03-29T11:38:44.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.893+0000 7f93b43d6640 1 -- start start 2024-03-29T11:38:44.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.893+0000 7f93b43d6640 10 monclient: init 2024-03-29T11:38:44.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.894+0000 7f93b43d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.894+0000 7f93b43d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.894+0000 7f93b43d6640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac0b4330 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.894+0000 7f93b43d6640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac1470b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.894+0000 7f93b43d6640 10 monclient(hunting): picked mon.noname-a con 0x7f93ac1439b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.894+0000 7f93b43d6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac06f2a0 con 0x7f93ac1439b0 2024-03-29T11:38:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.894+0000 7f93b43d6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac146b90 con 0x7f93ac0b4330 2024-03-29T11:38:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.894+0000 7f93b43d6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac0b2ca0 con 0x7f93ac1470b0 2024-03-29T11:38:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.894+0000 7f93b43d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.894+0000 7f93b43d6640 10 monclient(hunting): authenticate will time out at 1107.021793s 2024-03-29T11:38:44.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.901+0000 7f93b1971640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f93ac1470b0 0x7f93ac0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:45345/0 (socket says 172.21.15.155:45345) 2024-03-29T11:38:44.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.901+0000 7f93b1971640 1 -- 172.21.15.155:0/399923116 learned_addr learned my addr 172.21.15.155:0/399923116 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:44.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.901+0000 7f93b1170640 1 -- 172.21.15.155:0/399923116 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081810138 0 0) 0x7f93ac0b2ca0 con 0x7f93ac1470b0 2024-03-29T11:38:44.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.901+0000 7f93b1170640 10 monclient(hunting): _negotiate 2024-03-29T11:38:44.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.901+0000 7f93b1170640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:44.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.901+0000 7f93b1170640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:44.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.901+0000 7f93b1170640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:38:44.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.901+0000 7f93b1170640 1 -- 172.21.15.155:0/399923116 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9398003520 con 0x7f93ac1470b0 2024-03-29T11:38:44.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.901+0000 7f93b1170640 1 -- 172.21.15.155:0/399923116 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2129645743 0 0) 0x7f9398003520 con 0x7f93ac1470b0 2024-03-29T11:38:44.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.901+0000 7f93b1170640 1 -- 172.21.15.155:0/399923116 >> v1:172.21.15.183:6789/0 conn(0x7f93ac0b4330 legacy=0x7f93ac0b4710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:44.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 1 -- 172.21.15.155:0/399923116 >> v1:172.21.15.155:6789/0 conn(0x7f93ac1439b0 legacy=0x7f93ac151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 10 monclient: _finish_hunting 0 2024-03-29T11:38:44.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 1 monclient: found mon.noname-b 2024-03-29T11:38:44.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:44.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:38:44.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 1 -- 172.21.15.155:0/399923116 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac06e0c0 con 0x7f93ac1470b0 2024-03-29T11:38:44.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 10 monclient: _finish_auth 0 2024-03-29T11:38:44.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 10 monclient: _check_auth_tickets 2024-03-29T11:38:44.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 1 -- 172.21.15.155:0/399923116 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f939c001690 con 0x7f93ac1470b0 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.904+0000 7f93b1170640 10 monclient: dump: 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:44.906 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:44.907 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:44.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.906+0000 7f93b43d6640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:38:44.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.906+0000 7f93b43d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:44.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.907+0000 7f93b1170640 1 -- 172.21.15.155:0/399923116 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f939c0018f0 con 0x7f93ac1470b0 2024-03-29T11:38:44.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.907+0000 7f93b1170640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:44.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.907+0000 7f93b1170640 1 -- 172.21.15.155:0/399923116 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f939c001d40 con 0x7f93ac1470b0 2024-03-29T11:38:44.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.907+0000 7f93b1170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:44.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.907+0000 7f93b1170640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:44.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.907+0000 7f93b1170640 10 monclient: dump: 2024-03-29T11:38:44.909 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.907+0000 7f93b43d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.907+0000 7f93b43d6640 4 set_mon_vals no callback set 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.908+0000 7f93b43d6640 10 monclient: shutdown 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.908+0000 7f93b43d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:44.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.908+0000 7f93b43d6640 1 -- 172.21.15.155:0/399923116 >> v1:172.21.15.155:6790/0 conn(0x7f93ac1470b0 legacy=0x7f93ac0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.910+0000 7f93b43d6640 1 -- 172.21.15.155:0/399923116 shutdown_connections 2024-03-29T11:38:44.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.910+0000 7f93b43d6640 1 -- 172.21.15.155:0/399923116 >> 172.21.15.155:0/399923116 conn(0x7f93ac066a00 msgr2=0x7f93ac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:44.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.910+0000 7f93b43d6640 1 -- 172.21.15.155:0/399923116 shutdown_connections 2024-03-29T11:38:44.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.910+0000 7f93b43d6640 1 -- 172.21.15.155:0/399923116 wait complete. 2024-03-29T11:38:44.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.912+0000 7f93b43d6640 10 monclient: build_initial_monmap 2024-03-29T11:38:44.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.912+0000 7f93b43d6640 10 monclient: monmap: 2024-03-29T11:38:44.913 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:44.913 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:44.913 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:38:44.913 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:38:44.913 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:44.914 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:44.914 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:44.914 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:38:44.914 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:38:44.914 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:44.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.913+0000 7f93b43d6640 1 librados: starting msgr at 2024-03-29T11:38:44.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.913+0000 7f93b43d6640 1 librados: starting objecter 2024-03-29T11:38:44.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.913+0000 7f93b43d6640 1 Processor -- start 2024-03-29T11:38:44.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.915+0000 7f93b43d6640 1 -- start start 2024-03-29T11:38:44.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.915+0000 7f93b43d6640 1 librados: setting wanted keys 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.915+0000 7f93b43d6640 1 librados: calling monclient init 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.915+0000 7f93b43d6640 10 monclient: init 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.916+0000 7f93b43d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.916+0000 7f93b43d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.916+0000 7f93b43d6640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac0b4330 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.916+0000 7f93b43d6640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac1439b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.916+0000 7f93b43d6640 10 monclient(hunting): picked mon.noname-a con 0x7f93ac1470b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.916+0000 7f93b43d6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac11f770 con 0x7f93ac1470b0 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.916+0000 7f93b43d6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac11f900 con 0x7f93ac1439b0 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.916+0000 7f93b43d6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac1466c0 con 0x7f93ac0b4330 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.916+0000 7f93b43d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:44.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.916+0000 7f93b43d6640 10 monclient(hunting): authenticate will time out at 1107.043428s 2024-03-29T11:38:44.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.920+0000 7f93b2973640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f93ac1470b0 0x7f93ac1df550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:45703/0 (socket says 172.21.15.155:45703) 2024-03-29T11:38:44.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.920+0000 7f93b2973640 1 -- 172.21.15.155:0/2925663923 learned_addr learned my addr 172.21.15.155:0/2925663923 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:44.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.920+0000 7f93a2ffd640 10 client.?.objecter ms_handle_connect 0x7f93ac1470b0 2024-03-29T11:38:44.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.920+0000 7f93a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:44.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.922+0000 7f93a2ffd640 10 client.?.objecter ms_handle_connect 0x7f93ac1439b0 2024-03-29T11:38:44.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.922+0000 7f93a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:44.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.922+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194165548 0 0) 0x7f93ac11f770 con 0x7f93ac1470b0 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9390004eb0 con 0x7f93ac1470b0 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306956614 0 0) 0x7f93ac11f900 con 0x7f93ac1439b0 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93ac11f770 con 0x7f93ac1439b0 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 302354395 0 0) 0x7f93ac11f770 con 0x7f93ac1439b0 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93ac11f900 con 0x7f93ac1439b0 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f939c0014b0 con 0x7f93ac1439b0 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:44.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.923+0000 7f93a2ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:44.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 355021998 0 0) 0x7f9390004eb0 con 0x7f93ac1470b0 2024-03-29T11:38:44.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93ac11f770 con 0x7f93ac1470b0 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f93a4001910 con 0x7f93ac1470b0 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 10 client.?.objecter ms_handle_connect 0x7f93ac0b4330 2024-03-29T11:38:44.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 344892692 0 0) 0x7f93ac11f900 con 0x7f93ac1439b0 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 >> v1:172.21.15.155:6790/0 conn(0x7f93ac0b4330 legacy=0x7f93ac1406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 >> v1:172.21.15.155:6789/0 conn(0x7f93ac1470b0 legacy=0x7f93ac1df550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 1 monclient: found mon.b 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac146850 con 0x7f93ac1439b0 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93b43d6640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93b43d6640 10 monclient: _renew_subs 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93b43d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.924+0000 7f93b43d6640 1 -- 172.21.15.155:0/2925663923 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ac140e10 con 0x7f93ac1439b0 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.925+0000 7f93b43d6640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.925+0000 7f93b43d6640 10 monclient: _renew_subs 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.925+0000 7f93b43d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.925+0000 7f93b43d6640 1 -- 172.21.15.155:0/2925663923 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ac120930 con 0x7f93ac1439b0 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.925+0000 7f93b43d6640 1 librados: init done 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.925+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f939c001270 con 0x7f93ac1439b0 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.925+0000 7f93a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.925+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f939c004c70 con 0x7f93ac1439b0 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.925+0000 7f93a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.926+0000 7f93a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.926+0000 7f93a2ffd640 10 monclient: dump: 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.926+0000 7f93b43d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.926+0000 7f93b43d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.926+0000 7f93b43d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.926+0000 7f93b43d6640 1 -- 172.21.15.155:0/2925663923 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ac075f10 con 0x7f93ac1439b0 2024-03-29T11:38:44.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.927+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f939c01acc0 con 0x7f93ac1439b0 2024-03-29T11:38:44.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.928+0000 7f93b2172640 10 monclient: get_auth_request con 0x7f939006e1d0 auth_method 0 2024-03-29T11:38:44.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.928+0000 7f93a2ffd640 10 client.4378.objecter ms_handle_connect 0x7f939006e1d0 2024-03-29T11:38:44.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.928+0000 7f93a3fff640 4 set_mon_vals no callback set 2024-03-29T11:38:44.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.928+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f939c001610 con 0x7f93ac1439b0 2024-03-29T11:38:44.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.928+0000 7f93a2ffd640 10 client.4378.objecter ms_dispatch 0x7f93ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:44.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.928+0000 7f93a2ffd640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:44.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.928+0000 7f93a2ffd640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:44.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.929+0000 7f93a2ffd640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-29T11:38:44.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.935+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f939c050af0 con 0x7f93ac1439b0 2024-03-29T11:38:44.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.935+0000 7f93a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:44.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:44.935+0000 7f93a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:45.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.265+0000 7f93b43d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:45.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.265+0000 7f93b43d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:45.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.265+0000 7f93b43d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:45.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.265+0000 7f93b43d6640 1 -- 172.21.15.155:0/2925663923 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93ac141770 con 0x7f93ac1439b0 2024-03-29T11:38:45.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.266+0000 7f93a2ffd640 1 -- 172.21.15.155:0/2925663923 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f939c050af0 con 0x7f93ac1439b0 2024-03-29T11:38:45.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.266+0000 7f93a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:45.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.266+0000 7f93a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:45.268 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:38:45.268 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:45.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.271+0000 7f93b43d6640 10 librados: watch_flush enter 2024-03-29T11:38:45.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.271+0000 7f93b43d6640 10 librados: watch_flush exit 2024-03-29T11:38:45.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.271+0000 7f93b43d6640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-29T11:38:45.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.271+0000 7f93b43d6640 10 client.4378.objecter successfully canceled tick 2024-03-29T11:38:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.271+0000 7f93b43d6640 1 -- 172.21.15.155:0/2925663923 >> v1:172.21.15.155:6816/1760814708 conn(0x7f939006e1d0 legacy=0x7f9390070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.272+0000 7f93b43d6640 10 monclient: shutdown 2024-03-29T11:38:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.272+0000 7f93b43d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.272+0000 7f93b43d6640 1 -- 172.21.15.155:0/2925663923 >> v1:172.21.15.183:6789/0 conn(0x7f93ac1439b0 legacy=0x7f93ac1dee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:45.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.274+0000 7f93b43d6640 1 -- 172.21.15.155:0/2925663923 shutdown_connections 2024-03-29T11:38:45.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.274+0000 7f93b43d6640 1 -- 172.21.15.155:0/2925663923 >> 172.21.15.155:0/2925663923 conn(0x7f93ac066a00 msgr2=0x7f93ac1ce050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:45.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.274+0000 7f93b43d6640 1 -- 172.21.15.155:0/2925663923 shutdown_connections 2024-03-29T11:38:45.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.274+0000 7f93b43d6640 1 -- 172.21.15.155:0/2925663923 wait complete. 2024-03-29T11:38:45.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:45.274+0000 7f93b43d6640 1 librados: shutdown 2024-03-29T11:38:45.292 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:52.099 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:52.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.269+0000 7f8e074d6640 10 monclient: get_monmap_and_config 2024-03-29T11:38:52.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.269+0000 7f8e074d6640 10 monclient: build_initial_monmap 2024-03-29T11:38:52.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.269+0000 7f8e074d6640 10 monclient: monmap: 2024-03-29T11:38:52.269 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:52.269 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:52.269 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:38:52.269512+0000 2024-03-29T11:38:52.269 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:38:52.269512+0000 2024-03-29T11:38:52.269 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:52.269 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:52.270 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:52.270 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:38:52.270 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:38:52.270 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:52.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.272+0000 7f8e074d6640 1 Processor -- start 2024-03-29T11:38:52.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.272+0000 7f8e074d6640 1 -- start start 2024-03-29T11:38:52.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.272+0000 7f8e074d6640 10 monclient: init 2024-03-29T11:38:52.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.273+0000 7f8e074d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:52.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.273+0000 7f8e074d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:52.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.273+0000 7f8e074d6640 10 monclient(hunting): picked mon.noname-b con 0x7f8e001422b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:52.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.274+0000 7f8e074d6640 10 monclient(hunting): picked mon.noname-a con 0x7f8e00142690 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:52.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.274+0000 7f8e074d6640 10 monclient(hunting): picked mon.noname-c con 0x7f8e000b17d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:52.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.274+0000 7f8e074d6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e00142c80 con 0x7f8e00142690 2024-03-29T11:38:52.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.274+0000 7f8e074d6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e000aae40 con 0x7f8e000b17d0 2024-03-29T11:38:52.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.274+0000 7f8e074d6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0013fbd0 con 0x7f8e001422b0 2024-03-29T11:38:52.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.274+0000 7f8e074d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.274+0000 7f8e074d6640 10 monclient(hunting): authenticate will time out at 1114.400264s 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.274+0000 7f8e05272640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f8e001422b0 0x7f8e0013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:43340/0 (socket says 172.21.15.155:43340) 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.274+0000 7f8e05272640 1 -- 172.21.15.155:0/331790644 learned_addr learned my addr 172.21.15.155:0/331790644 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.275+0000 7f8df7fff640 1 -- 172.21.15.155:0/331790644 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871656179 0 0) 0x7f8e0013fbd0 con 0x7f8e001422b0 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.275+0000 7f8df7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.275+0000 7f8df7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.275+0000 7f8df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.275+0000 7f8df7fff640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.275+0000 7f8df7fff640 1 -- 172.21.15.155:0/331790644 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dec003520 con 0x7f8e001422b0 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 1 -- 172.21.15.155:0/331790644 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 621795772 0 0) 0x7f8dec003520 con 0x7f8e001422b0 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 1 -- 172.21.15.155:0/331790644 >> v1:172.21.15.183:6789/0 conn(0x7f8e000b17d0 legacy=0x7f8e000b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 1 -- 172.21.15.155:0/331790644 >> v1:172.21.15.155:6789/0 conn(0x7f8e00142690 legacy=0x7f8e000b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:52.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 1 monclient: found mon.noname-b 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 1 -- 172.21.15.155:0/331790644 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e00151940 con 0x7f8e001422b0 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 1 -- 172.21.15.155:0/331790644 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8dfc001690 con 0x7f8e001422b0 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8df7fff640 10 monclient: dump: 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:52.282 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8e074d6640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.276+0000 7f8e074d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.277+0000 7f8df7fff640 1 -- 172.21.15.155:0/331790644 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dfc0017f0 con 0x7f8e001422b0 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.277+0000 7f8df7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.277+0000 7f8df7fff640 1 -- 172.21.15.155:0/331790644 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8dfc001c40 con 0x7f8e001422b0 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.277+0000 7f8df7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.277+0000 7f8df7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.277+0000 7f8df7fff640 10 monclient: dump: 2024-03-29T11:38:52.283 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.277+0000 7f8e074d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.277+0000 7f8e074d6640 4 set_mon_vals no callback set 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.278+0000 7f8e074d6640 10 monclient: shutdown 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.278+0000 7f8e074d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.278+0000 7f8e074d6640 1 -- 172.21.15.155:0/331790644 >> v1:172.21.15.155:6790/0 conn(0x7f8e001422b0 legacy=0x7f8e0013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.279+0000 7f8e074d6640 1 -- 172.21.15.155:0/331790644 shutdown_connections 2024-03-29T11:38:52.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.279+0000 7f8e074d6640 1 -- 172.21.15.155:0/331790644 >> 172.21.15.155:0/331790644 conn(0x7f8e00066a00 msgr2=0x7f8e00066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:52.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.280+0000 7f8e074d6640 1 -- 172.21.15.155:0/331790644 shutdown_connections 2024-03-29T11:38:52.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.280+0000 7f8e074d6640 1 -- 172.21.15.155:0/331790644 wait complete. 2024-03-29T11:38:52.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.281+0000 7f8e074d6640 10 monclient: build_initial_monmap 2024-03-29T11:38:52.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.281+0000 7f8e074d6640 10 monclient: monmap: 2024-03-29T11:38:52.286 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:52.286 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:52.286 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:38:52.286 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.281+0000 7f8e074d6640 1 librados: starting msgr at 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.282+0000 7f8e074d6640 1 librados: starting objecter 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.282+0000 7f8e074d6640 1 Processor -- start 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.282+0000 7f8e074d6640 1 -- start start 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.282+0000 7f8e074d6640 1 librados: setting wanted keys 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.283+0000 7f8e074d6640 1 librados: calling monclient init 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.283+0000 7f8e074d6640 10 monclient: init 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.284+0000 7f8e074d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:52.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.284+0000 7f8e074d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:52.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.284+0000 7f8e074d6640 10 monclient(hunting): picked mon.noname-b con 0x7f8e000b17d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:52.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.284+0000 7f8e074d6640 10 monclient(hunting): picked mon.noname-c con 0x7f8e001422b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:52.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.284+0000 7f8e074d6640 10 monclient(hunting): picked mon.noname-a con 0x7f8e00142690 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:52.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.284+0000 7f8e074d6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e00132040 con 0x7f8e00142690 2024-03-29T11:38:52.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.284+0000 7f8e074d6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e000ac730 con 0x7f8e000b17d0 2024-03-29T11:38:52.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.284+0000 7f8e074d6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e000aca10 con 0x7f8e001422b0 2024-03-29T11:38:52.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.284+0000 7f8e074d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:52.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.284+0000 7f8e074d6640 10 monclient(hunting): authenticate will time out at 1114.410548s 2024-03-29T11:38:52.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.288+0000 7f8e05a73640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f8e00142690 0x7f8e001e1c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:43185/0 (socket says 172.21.15.155:43185) 2024-03-29T11:38:52.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.288+0000 7f8e05a73640 1 -- 172.21.15.155:0/1370198472 learned_addr learned my addr 172.21.15.155:0/1370198472 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:52.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.288+0000 7f8df5ffb640 10 client.?.objecter ms_handle_connect 0x7f8e00142690 2024-03-29T11:38:52.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.288+0000 7f8df5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:52.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.289+0000 7f8df5ffb640 10 client.?.objecter ms_handle_connect 0x7f8e001422b0 2024-03-29T11:38:52.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.289+0000 7f8df5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:52.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 10 client.?.objecter ms_handle_connect 0x7f8e000b17d0 2024-03-29T11:38:52.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:52.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864019691 0 0) 0x7f8e00132040 con 0x7f8e00142690 2024-03-29T11:38:52.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:52.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:52.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:52.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:38:52.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de4004eb0 con 0x7f8e00142690 2024-03-29T11:38:52.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529584818 0 0) 0x7f8e000ac730 con 0x7f8e000b17d0 2024-03-29T11:38:52.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:52.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:52.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:52.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:38:52.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.290+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de4006230 con 0x7f8e000b17d0 2024-03-29T11:38:52.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 147960447 0 0) 0x7f8de4006230 con 0x7f8e000b17d0 2024-03-29T11:38:52.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e000ac730 con 0x7f8e000b17d0 2024-03-29T11:38:52.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8dfc001b00 con 0x7f8e000b17d0 2024-03-29T11:38:52.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:52.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:52.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:52.295 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2076870101 0 0) 0x7f8de4004eb0 con 0x7f8e00142690 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8de4006230 con 0x7f8e00142690 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324299346 0 0) 0x7f8e000aca10 con 0x7f8e001422b0 2024-03-29T11:38:52.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de4006d50 con 0x7f8e001422b0 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8df8001b30 con 0x7f8e00142690 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:52.297 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3843255591 0 0) 0x7f8e000ac730 con 0x7f8e000b17d0 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 >> v1:172.21.15.155:6790/0 conn(0x7f8e001422b0 legacy=0x7f8e00130420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 >> v1:172.21.15.155:6789/0 conn(0x7f8e00142690 legacy=0x7f8e001e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 monclient: found mon.b 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e000accf0 con 0x7f8e000b17d0 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.291+0000 7f8df5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dfc001ca0 con 0x7f8e000b17d0 2024-03-29T11:38:52.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8df5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8dfc004d90 con 0x7f8e000b17d0 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8df5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8df5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8df5ffb640 10 monclient: dump: 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:52.299 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:52.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8df77fe640 4 set_mon_vals no callback set 2024-03-29T11:38:52.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8e074d6640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:38:52.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8e074d6640 10 monclient: _renew_subs 2024-03-29T11:38:52.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8e074d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:52.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8e074d6640 1 -- 172.21.15.155:0/1370198472 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e00132840 con 0x7f8e000b17d0 2024-03-29T11:38:52.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8e074d6640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:52.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8e074d6640 10 monclient: _renew_subs 2024-03-29T11:38:52.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8e074d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:52.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8e074d6640 1 -- 172.21.15.155:0/1370198472 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e00132af0 con 0x7f8e000b17d0 2024-03-29T11:38:52.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.292+0000 7f8e074d6640 1 librados: init done 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.294+0000 7f8e074d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.294+0000 7f8e074d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.294+0000 7f8e074d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.294+0000 7f8e074d6640 1 -- 172.21.15.155:0/1370198472 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e00131e10 con 0x7f8e000b17d0 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.296+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f8dfc01ad60 con 0x7f8e000b17d0 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.296+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f8dfc084a80 con 0x7f8e000b17d0 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.296+0000 7f8df5ffb640 10 client.4381.objecter ms_dispatch 0x7f8e00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.296+0000 7f8df5ffb640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.296+0000 7f8df5ffb640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.296+0000 7f8df5ffb640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.296+0000 7f8e04a71640 10 monclient: get_auth_request con 0x7f8de406e470 auth_method 0 2024-03-29T11:38:52.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.297+0000 7f8df5ffb640 10 client.4381.objecter ms_handle_connect 0x7f8de406e470 2024-03-29T11:38:52.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.302+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e00131e10 con 0x7f8e000b17d0 2024-03-29T11:38:52.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.302+0000 7f8df5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:52.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.302+0000 7f8df5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:52.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.635+0000 7f8e074d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:52.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.635+0000 7f8e074d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:52.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.635+0000 7f8e074d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:52.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.635+0000 7f8e074d6640 1 -- 172.21.15.155:0/1370198472 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e00131e10 con 0x7f8e000b17d0 2024-03-29T11:38:52.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.636+0000 7f8df5ffb640 1 -- 172.21.15.155:0/1370198472 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f8e00131e10 con 0x7f8e000b17d0 2024-03-29T11:38:52.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.636+0000 7f8df5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:52.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.636+0000 7f8df5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:52.637 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:38:52.637 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:52.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.640+0000 7f8e074d6640 10 librados: watch_flush enter 2024-03-29T11:38:52.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.641+0000 7f8e074d6640 10 librados: watch_flush exit 2024-03-29T11:38:52.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.641+0000 7f8e074d6640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-29T11:38:52.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.641+0000 7f8e074d6640 10 client.4381.objecter successfully canceled tick 2024-03-29T11:38:52.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.641+0000 7f8e074d6640 1 -- 172.21.15.155:0/1370198472 >> v1:172.21.15.155:6816/1760814708 conn(0x7f8de406e470 legacy=0x7f8de4070930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:52.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.641+0000 7f8e074d6640 10 monclient: shutdown 2024-03-29T11:38:52.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.641+0000 7f8e074d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:52.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.641+0000 7f8e074d6640 1 -- 172.21.15.155:0/1370198472 >> v1:172.21.15.183:6789/0 conn(0x7f8e000b17d0 legacy=0x7f8e0012fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:52.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.642+0000 7f8e074d6640 1 -- 172.21.15.155:0/1370198472 shutdown_connections 2024-03-29T11:38:52.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.642+0000 7f8e074d6640 1 -- 172.21.15.155:0/1370198472 >> 172.21.15.155:0/1370198472 conn(0x7f8e00066a00 msgr2=0x7f8e00066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:52.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.643+0000 7f8e074d6640 1 -- 172.21.15.155:0/1370198472 shutdown_connections 2024-03-29T11:38:52.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.644+0000 7f8e074d6640 1 -- 172.21.15.155:0/1370198472 wait complete. 2024-03-29T11:38:52.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:52.644+0000 7f8e074d6640 1 librados: shutdown 2024-03-29T11:38:52.664 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:59.471 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:59.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.625+0000 7fe0f359e640 10 monclient: get_monmap_and_config 2024-03-29T11:38:59.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.626+0000 7fe0f359e640 10 monclient: build_initial_monmap 2024-03-29T11:38:59.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.626+0000 7fe0f359e640 10 monclient: monmap: 2024-03-29T11:38:59.626 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:59.626 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:59.627 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:38:59.626445+0000 2024-03-29T11:38:59.627 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:38:59.626445+0000 2024-03-29T11:38:59.627 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:59.627 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:59.627 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:59.627 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:38:59.627 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:38:59.627 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:59.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.626+0000 7fe0f359e640 1 Processor -- start 2024-03-29T11:38:59.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.627+0000 7fe0f359e640 1 -- start start 2024-03-29T11:38:59.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.627+0000 7fe0f359e640 10 monclient: init 2024-03-29T11:38:59.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:59.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:59.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f359e640 10 monclient(hunting): picked mon.noname-b con 0x7fe0f40b2800 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:59.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f359e640 10 monclient(hunting): picked mon.noname-a con 0x7fe0f40a9c00 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:59.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f359e640 10 monclient(hunting): picked mon.noname-c con 0x7fe0f40b38c0 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:59.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f359e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f4141eb0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f259c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fe0f40b2800 0x7fe0f4142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33760/0 (socket says 172.21.15.155:33760) 2024-03-29T11:38:59.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f259c640 1 -- 172.21.15.155:0/4049566622 learned_addr learned my addr 172.21.15.155:0/4049566622 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f359e640 1 -- 172.21.15.155:0/4049566622 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f40b22d0 con 0x7fe0f40b38c0 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f359e640 1 -- 172.21.15.155:0/4049566622 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f4142d30 con 0x7fe0f40b2800 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f359e640 10 monclient(hunting): authenticate will time out at 1121.754252s 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f159a640 1 -- 172.21.15.155:0/4049566622 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365521472 0 0) 0x7fe0f4141eb0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f159a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f159a640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f159a640 1 -- 172.21.15.155:0/4049566622 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0dc003520 con 0x7fe0f40a9c00 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f159a640 1 -- 172.21.15.155:0/4049566622 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795092327 0 0) 0x7fe0f4142d30 con 0x7fe0f40b2800 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.628+0000 7fe0f159a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:59.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:38:59.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 1 -- 172.21.15.155:0/4049566622 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f4141eb0 con 0x7fe0f40b2800 2024-03-29T11:38:59.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 1 -- 172.21.15.155:0/4049566622 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1569009034 0 0) 0x7fe0dc003520 con 0x7fe0f40a9c00 2024-03-29T11:38:59.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 1 -- 172.21.15.155:0/4049566622 >> v1:172.21.15.155:6790/0 conn(0x7fe0f40b2800 legacy=0x7fe0f4142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 1 -- 172.21.15.155:0/4049566622 >> v1:172.21.15.183:6789/0 conn(0x7fe0f40b38c0 legacy=0x7fe0f414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:59.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 1 monclient: found mon.noname-a 2024-03-29T11:38:59.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:59.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:38:59.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 1 -- 172.21.15.155:0/4049566622 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f40aaf90 con 0x7fe0f40a9c00 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 10 monclient: _finish_auth 0 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f359e640 5 monclient: authenticate success, global_id 4389 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 1 -- 172.21.15.155:0/4049566622 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe0e4001690 con 0x7fe0f40a9c00 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f159a640 10 monclient: dump: 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:59.638 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.629+0000 7fe0f359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.630+0000 7fe0f159a640 1 -- 172.21.15.155:0/4049566622 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0e40018b0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.630+0000 7fe0f159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.630+0000 7fe0f159a640 1 -- 172.21.15.155:0/4049566622 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe0e4001bc0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.630+0000 7fe0f359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.630+0000 7fe0f359e640 4 set_mon_vals no callback set 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.630+0000 7fe0f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.630+0000 7fe0f159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.630+0000 7fe0f159a640 10 monclient: dump: 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:59.639 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.630+0000 7fe0f359e640 10 monclient: shutdown 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.630+0000 7fe0f359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.630+0000 7fe0f359e640 1 -- 172.21.15.155:0/4049566622 >> v1:172.21.15.155:6789/0 conn(0x7fe0f40a9c00 legacy=0x7fe0f413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.631+0000 7fe0f359e640 1 -- 172.21.15.155:0/4049566622 shutdown_connections 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.631+0000 7fe0f359e640 1 -- 172.21.15.155:0/4049566622 >> 172.21.15.155:0/4049566622 conn(0x7fe0f4066a00 msgr2=0x7fe0f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.631+0000 7fe0f359e640 1 -- 172.21.15.155:0/4049566622 shutdown_connections 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.631+0000 7fe0f359e640 1 -- 172.21.15.155:0/4049566622 wait complete. 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.632+0000 7fe0f359e640 10 monclient: build_initial_monmap 2024-03-29T11:38:59.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.632+0000 7fe0f359e640 10 monclient: monmap: 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.632+0000 7fe0f359e640 1 librados: starting msgr at 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.632+0000 7fe0f359e640 1 librados: starting objecter 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.633+0000 7fe0f359e640 1 Processor -- start 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.633+0000 7fe0f359e640 1 -- start start 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.633+0000 7fe0f359e640 1 librados: setting wanted keys 2024-03-29T11:38:59.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.633+0000 7fe0f359e640 1 librados: calling monclient init 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.633+0000 7fe0f359e640 10 monclient: init 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f359e640 10 monclient(hunting): picked mon.noname-b con 0x7fe0f40a9c00 addr v1:172.21.15.183:6789/0 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f359e640 10 monclient(hunting): picked mon.noname-c con 0x7fe0f40b2800 addr v1:172.21.15.155:6790/0 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f1d9b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fe0f40b2800 0x7fe0f41d7510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:34414/0 (socket says 172.21.15.155:34414) 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f1d9b640 1 -- 172.21.15.155:0/3983375557 learned_addr learned my addr 172.21.15.155:0/3983375557 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f359e640 10 monclient(hunting): picked mon.noname-a con 0x7fe0f40b38c0 addr v1:172.21.15.155:6789/0 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f359e640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f41d7080 con 0x7fe0f40b38c0 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f359e640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f413c720 con 0x7fe0f40a9c00 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f359e640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f40ad010 con 0x7fe0f40b2800 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0e2ffd640 10 client.?.objecter ms_handle_connect 0x7fe0f40b2800 2024-03-29T11:38:59.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0e2ffd640 10 client.?.objecter ms_handle_connect 0x7fe0f40a9c00 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0f359e640 10 monclient(hunting): authenticate will time out at 1121.760488s 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0e2ffd640 10 client.?.objecter ms_handle_connect 0x7fe0f40b38c0 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.634+0000 7fe0e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478199539 0 0) 0x7fe0f41d7080 con 0x7fe0f40b38c0 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d4004eb0 con 0x7fe0f40b38c0 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715972347 0 0) 0x7fe0f413c720 con 0x7fe0f40a9c00 2024-03-29T11:38:59.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f41d7080 con 0x7fe0f40a9c00 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426805160 0 0) 0x7fe0f40ad010 con 0x7fe0f40b2800 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f413c720 con 0x7fe0f40b2800 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4000673679 0 0) 0x7fe0f41d7080 con 0x7fe0f40a9c00 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0f40ad010 con 0x7fe0f40a9c00 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe0e80018b0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:59.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.635+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3199384575 0 0) 0x7fe0f413c720 con 0x7fe0f40b2800 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0f41d7080 con 0x7fe0f40b2800 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3402803621 0 0) 0x7fe0d4004eb0 con 0x7fe0f40b38c0 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0f413c720 con 0x7fe0f40b38c0 2024-03-29T11:38:59.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 459775772 0 0) 0x7fe0f40ad010 con 0x7fe0f40a9c00 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 >> v1:172.21.15.155:6790/0 conn(0x7fe0f40b2800 legacy=0x7fe0f41d7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 >> v1:172.21.15.155:6789/0 conn(0x7fe0f40b38c0 legacy=0x7fe0f40b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 1 monclient: found mon.b 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f40ad2f0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0f359e640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0f359e640 10 monclient: _renew_subs 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0f359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0f359e640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f40abcd0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0f359e640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:59.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0f359e640 10 monclient: _renew_subs 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0f359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0f359e640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f40abf80 con 0x7fe0f40a9c00 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.636+0000 7fe0f359e640 1 librados: init done 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.637+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0e80047e0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.637+0000 7fe0e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.637+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe0e8004af0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.637+0000 7fe0e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.637+0000 7fe0e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.637+0000 7fe0e2ffd640 10 monclient: dump: 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:59.647 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:38:59.648 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:38:59.648 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:38:59.648 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:38:59.648 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:38:59.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.637+0000 7fe0f0d99640 4 set_mon_vals no callback set 2024-03-29T11:38:59.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.638+0000 7fe0e0ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:59.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.638+0000 7fe0e0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:59.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.638+0000 7fe0e0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:59.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.638+0000 7fe0e0ff9640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0cc005080 con 0x7fe0f40a9c00 2024-03-29T11:38:59.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.639+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fe0e801aac0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.640+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fe0e80849f0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.640+0000 7fe0e2ffd640 10 client.4384.objecter ms_dispatch 0x7fe0f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:59.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.640+0000 7fe0e2ffd640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:59.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.640+0000 7fe0e2ffd640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:59.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.640+0000 7fe0e2ffd640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-29T11:38:59.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.640+0000 7fe0f1d9b640 10 monclient: get_auth_request con 0x7fe0d406e030 auth_method 0 2024-03-29T11:38:59.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.640+0000 7fe0e2ffd640 10 client.4384.objecter ms_handle_connect 0x7fe0d406e030 2024-03-29T11:38:59.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.648+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0e80502d0 con 0x7fe0f40a9c00 2024-03-29T11:38:59.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.648+0000 7fe0e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:59.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.648+0000 7fe0e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:59.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.978+0000 7fe0e0ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:59.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.978+0000 7fe0e0ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:59.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.978+0000 7fe0e0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:38:59.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.978+0000 7fe0e0ff9640 1 -- 172.21.15.155:0/3983375557 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0cc005330 con 0x7fe0f40a9c00 2024-03-29T11:38:59.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.980+0000 7fe0e2ffd640 1 -- 172.21.15.155:0/3983375557 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fe0e8052200 con 0x7fe0f40a9c00 2024-03-29T11:38:59.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.980+0000 7fe0e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:59.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.980+0000 7fe0e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:59.982 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:38:59.982 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.983+0000 7fe0e0ff9640 10 librados: watch_flush enter 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 10 librados: watch_flush exit 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 10 client.4384.objecter successfully canceled tick 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 1 -- 172.21.15.155:0/3983375557 >> v1:172.21.15.155:6816/1760814708 conn(0x7fe0d406e030 legacy=0x7fe0d40704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 10 monclient: shutdown 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 1 -- 172.21.15.155:0/3983375557 >> v1:172.21.15.183:6789/0 conn(0x7fe0f40a9c00 legacy=0x7fe0f40ab5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 1 -- 172.21.15.155:0/3983375557 shutdown_connections 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 1 -- 172.21.15.155:0/3983375557 >> 172.21.15.155:0/3983375557 conn(0x7fe0f4066a00 msgr2=0x7fe0f413bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 1 -- 172.21.15.155:0/3983375557 shutdown_connections 2024-03-29T11:38:59.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 1 -- 172.21.15.155:0/3983375557 wait complete. 2024-03-29T11:38:59.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:38:59.984+0000 7fe0e0ff9640 1 librados: shutdown 2024-03-29T11:39:00.001 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:06.807 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:06.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.959+0000 7f2ffe8d9640 10 monclient: get_monmap_and_config 2024-03-29T11:39:06.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.959+0000 7f2ffe8d9640 10 monclient: build_initial_monmap 2024-03-29T11:39:06.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.959+0000 7f2ffe8d9640 10 monclient: monmap: 2024-03-29T11:39:06.960 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:06.960 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:06.961 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:39:06.960089+0000 2024-03-29T11:39:06.961 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:39:06.960089+0000 2024-03-29T11:39:06.961 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:06.961 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:06.961 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:06.961 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:39:06.961 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:39:06.961 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:06.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.960+0000 7f2ffe8d9640 1 Processor -- start 2024-03-29T11:39:06.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.960+0000 7f2ffe8d9640 1 -- start start 2024-03-29T11:39:06.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.960+0000 7f2ffe8d9640 10 monclient: init 2024-03-29T11:39:06.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.961+0000 7f2ffe8d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:06.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.961+0000 7f2ffe8d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:06.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.961+0000 7f2ffe8d9640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff80b4330 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:06.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.961+0000 7f2ffe8d9640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff81470b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:06.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.961+0000 7f2ffe8d9640 10 monclient(hunting): picked mon.noname-b con 0x7f2ff81439b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:06.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.961+0000 7f2ffe8d9640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff806f2a0 con 0x7f2ff80b4330 2024-03-29T11:39:06.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.961+0000 7f2ffe8d9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff8146b90 con 0x7f2ff81470b0 2024-03-29T11:39:06.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.962+0000 7f2ffe8d9640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff80b2ca0 con 0x7f2ff81439b0 2024-03-29T11:39:06.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.962+0000 7f2ffe8d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:06.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.962+0000 7f2ff7fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f2ff80b4330 0x7f2ff80b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:33530/0 (socket says 172.21.15.155:33530) 2024-03-29T11:39:06.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.962+0000 7f2ff7fff640 1 -- 172.21.15.155:0/3265424831 learned_addr learned my addr 172.21.15.155:0/3265424831 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:06.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.962+0000 7f2ffe8d9640 10 monclient(hunting): authenticate will time out at 1129.087917s 2024-03-29T11:39:06.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.962+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226739445 0 0) 0x7f2ff806f2a0 con 0x7f2ff80b4330 2024-03-29T11:39:06.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.962+0000 7f2ff6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:06.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.962+0000 7f2ff6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:06.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.962+0000 7f2ff6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:06.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.962+0000 7f2ff6ffd640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:39:06.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.962+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe4003520 con 0x7f2ff80b4330 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379156989 0 0) 0x7f2ff8146b90 con 0x7f2ff81470b0 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff806f2a0 con 0x7f2ff81470b0 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668592732 0 0) 0x7f2ff80b2ca0 con 0x7f2ff81439b0 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff8146b90 con 0x7f2ff81439b0 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3451354645 0 0) 0x7f2fe4003520 con 0x7f2ff80b4330 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 >> v1:172.21.15.155:6790/0 conn(0x7f2ff81439b0 legacy=0x7f2ff8151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 >> v1:172.21.15.183:6789/0 conn(0x7f2ff81470b0 legacy=0x7f2ff80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.963+0000 7f2ff6ffd640 1 monclient: found mon.noname-a 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff806e0c0 con 0x7f2ff80b4330 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2fe8001a10 con 0x7f2ff80b4330 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 10 monclient: dump: 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:06.974 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ffe8d9640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ffe8d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fe8001c30 con 0x7f2ff80b4330 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 1 -- 172.21.15.155:0/3265424831 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2fe8001fc0 con 0x7f2ff80b4330 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:06.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ff6ffd640 10 monclient: dump: 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ffe8d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:06.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.964+0000 7f2ffe8d9640 4 set_mon_vals no callback set 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.965+0000 7f2ffe8d9640 10 monclient: shutdown 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.965+0000 7f2ffe8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.965+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/3265424831 >> v1:172.21.15.155:6789/0 conn(0x7f2ff80b4330 legacy=0x7f2ff80b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.965+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/3265424831 shutdown_connections 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.965+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/3265424831 >> 172.21.15.155:0/3265424831 conn(0x7f2ff8066a00 msgr2=0x7f2ff8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.965+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/3265424831 shutdown_connections 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.965+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/3265424831 wait complete. 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.966+0000 7f2ffe8d9640 10 monclient: build_initial_monmap 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.966+0000 7f2ffe8d9640 10 monclient: monmap: 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:06.977 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.967+0000 7f2ffe8d9640 1 librados: starting msgr at 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.967+0000 7f2ffe8d9640 1 librados: starting objecter 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.967+0000 7f2ffe8d9640 1 Processor -- start 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.967+0000 7f2ffe8d9640 1 -- start start 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.967+0000 7f2ffe8d9640 1 librados: setting wanted keys 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.967+0000 7f2ffe8d9640 1 librados: calling monclient init 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.967+0000 7f2ffe8d9640 10 monclient: init 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.968+0000 7f2ffe8d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.968+0000 7f2ffe8d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.968+0000 7f2ffe8d9640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff80b4330 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.968+0000 7f2ff7fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f2ff80b4330 0x7f2ff81db360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34112/0 (socket says 172.21.15.155:34112) 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.968+0000 7f2ff7fff640 1 -- 172.21.15.155:0/1093059814 learned_addr learned my addr 172.21.15.155:0/1093059814 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:06.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.968+0000 7f2ffe8d9640 10 monclient(hunting): picked mon.noname-b con 0x7f2ff81439b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.968+0000 7f2ffe8d9640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff81470b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.968+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff8121780 con 0x7f2ff80b4330 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.968+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff81dba70 con 0x7f2ff81439b0 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.968+0000 7f2ff4ff9640 10 client.?.objecter ms_handle_connect 0x7f2ff80b4330 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 client.?.objecter ms_handle_connect 0x7f2ff81439b0 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff811f770 con 0x7f2ff81470b0 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ffe8d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 client.?.objecter ms_handle_connect 0x7f2ff81470b0 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ffe8d9640 10 monclient(hunting): authenticate will time out at 1129.095076s 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772787780 0 0) 0x7f2ff811f770 con 0x7f2ff81470b0 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:39:06.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd8004eb0 con 0x7f2ff81470b0 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451519673 0 0) 0x7f2ff8121780 con 0x7f2ff80b4330 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.969+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff811f770 con 0x7f2ff80b4330 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4090051450 0 0) 0x7f2fd8004eb0 con 0x7f2ff81470b0 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ff8121780 con 0x7f2ff81470b0 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2ff0001a10 con 0x7f2ff81470b0 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:06.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient(hunting): dump: 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721196056 0 0) 0x7f2ff81dba70 con 0x7f2ff81439b0 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:06.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd8004eb0 con 0x7f2ff81439b0 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 536398675 0 0) 0x7f2ff8121780 con 0x7f2ff81470b0 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 >> v1:172.21.15.183:6789/0 conn(0x7f2ff81439b0 legacy=0x7f2ff81402d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 >> v1:172.21.15.155:6789/0 conn(0x7f2ff80b4330 legacy=0x7f2ff81db360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 1 monclient: found mon.c 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff811f900 con 0x7f2ff81470b0 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.970+0000 7f2ff4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ffe8d9640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:39:06.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ffe8d9640 10 monclient: _renew_subs 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ffe8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ff00035f0 con 0x7f2ff81470b0 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff81202d0 con 0x7f2ff81470b0 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ffe8d9640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ffe8d9640 10 monclient: _renew_subs 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ffe8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff8140bc0 con 0x7f2ff81470b0 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ffe8d9640 1 librados: init done 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ff4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2ff0003900 con 0x7f2ff81470b0 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ff4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ff4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.971+0000 7f2ff4ff9640 10 monclient: dump: 2024-03-29T11:39:06.983 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:06.984 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:06.984 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:06.984 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:06.984 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:06.984 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:06.984 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:06.984 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:06.984 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:06.984 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:06.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.972+0000 7f2ff67fc640 4 set_mon_vals no callback set 2024-03-29T11:39:06.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.972+0000 7f2ffe8d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.973+0000 7f2ffe8d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.973+0000 7f2ffe8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.973+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fdc005080 con 0x7f2ff81470b0 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.975+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f2ff00198d0 con 0x7f2ff81470b0 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.976+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f2ff00837d0 con 0x7f2ff81470b0 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.976+0000 7f2ff4ff9640 10 client.4403.objecter ms_dispatch 0x7f2ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.976+0000 7f2ff4ff9640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.976+0000 7f2ff4ff9640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.976+0000 7f2ff4ff9640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.976+0000 7f2ff7fff640 10 monclient: get_auth_request con 0x7f2fd806dea0 auth_method 0 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.976+0000 7f2ff4ff9640 10 client.4403.objecter ms_handle_connect 0x7f2fd806dea0 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.984+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ff004f0b0 con 0x7f2ff81470b0 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.984+0000 7f2ff4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:06.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:06.984+0000 7f2ff4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:07.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.363+0000 7f2ffe8d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:07.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.363+0000 7f2ffe8d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:07.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.363+0000 7f2ffe8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:07.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.363+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fdc005330 con 0x7f2ff81470b0 2024-03-29T11:39:07.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.364+0000 7f2ff4ff9640 1 -- 172.21.15.155:0/1093059814 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f2ff0050fe0 con 0x7f2ff81470b0 2024-03-29T11:39:07.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.364+0000 7f2ff4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:07.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.364+0000 7f2ff4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:07.365 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:39:07.365 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:07.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.368+0000 7f2ffe8d9640 10 librados: watch_flush enter 2024-03-29T11:39:07.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.368+0000 7f2ffe8d9640 10 librados: watch_flush exit 2024-03-29T11:39:07.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.368+0000 7f2ffe8d9640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T11:39:07.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.368+0000 7f2ffe8d9640 10 client.4403.objecter successfully canceled tick 2024-03-29T11:39:07.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.368+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 >> v1:172.21.15.155:6816/1760814708 conn(0x7f2fd806dea0 legacy=0x7f2fd8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:07.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.368+0000 7f2ffe8d9640 10 monclient: shutdown 2024-03-29T11:39:07.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.368+0000 7f2ffe8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:07.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.368+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 >> v1:172.21.15.155:6790/0 conn(0x7f2ff81470b0 legacy=0x7f2ff81e6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:07.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.368+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 shutdown_connections 2024-03-29T11:39:07.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.368+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 >> 172.21.15.155:0/1093059814 conn(0x7f2ff8066a00 msgr2=0x7f2ff8143100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:07.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.369+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 shutdown_connections 2024-03-29T11:39:07.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.369+0000 7f2ffe8d9640 1 -- 172.21.15.155:0/1093059814 wait complete. 2024-03-29T11:39:07.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:07.369+0000 7f2ffe8d9640 1 librados: shutdown 2024-03-29T11:39:07.383 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:14.190 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:14.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.344+0000 7f065773e640 10 monclient: get_monmap_and_config 2024-03-29T11:39:14.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.344+0000 7f065773e640 10 monclient: build_initial_monmap 2024-03-29T11:39:14.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.344+0000 7f065773e640 10 monclient: monmap: 2024-03-29T11:39:14.346 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:14.346 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:14.346 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:39:14.345848+0000 2024-03-29T11:39:14.346 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:39:14.345848+0000 2024-03-29T11:39:14.346 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:14.346 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:14.346 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:14.347 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:39:14.347 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:39:14.347 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:14.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.346+0000 7f065773e640 1 Processor -- start 2024-03-29T11:39:14.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.346+0000 7f065773e640 1 -- start start 2024-03-29T11:39:14.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.346+0000 7f065773e640 10 monclient: init 2024-03-29T11:39:14.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.347+0000 7f065773e640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:14.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.347+0000 7f065773e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:14.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.347+0000 7f065773e640 10 monclient(hunting): picked mon.noname-a con 0x7f06500b0770 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.347+0000 7f065773e640 10 monclient(hunting): picked mon.noname-c con 0x7f065014a700 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.347+0000 7f065773e640 10 monclient(hunting): picked mon.noname-b con 0x7f06500ac5a0 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.347+0000 7f065773e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06500aba70 con 0x7f06500b0770 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.347+0000 7f065773e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06500aecb0 con 0x7f065014a700 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.347+0000 7f065773e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06500af150 con 0x7f06500ac5a0 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.347+0000 7f065773e640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.347+0000 7f065773e640 10 monclient(hunting): authenticate will time out at 1136.474473s 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.350+0000 7f0655cdb640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f06500ac5a0 0x7f0650151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:49754/0 (socket says 172.21.15.155:49754) 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.350+0000 7f0655cdb640 1 -- 172.21.15.155:0/2745296485 learned_addr learned my addr 172.21.15.155:0/2745296485 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698122280 0 0) 0x7f06500aba70 con 0x7f06500b0770 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:39:14.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f063c003520 con 0x7f06500b0770 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515015994 0 0) 0x7f06500aecb0 con 0x7f065014a700 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f063c0048a0 con 0x7f065014a700 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141290848 0 0) 0x7f06500af150 con 0x7f06500ac5a0 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.351+0000 7f0647fff640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06500aecb0 con 0x7f06500ac5a0 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3137977820 0 0) 0x7f06500aecb0 con 0x7f06500ac5a0 2024-03-29T11:39:14.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 >> v1:172.21.15.183:6789/0 conn(0x7f065014a700 legacy=0x7f06500abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 >> v1:172.21.15.155:6789/0 conn(0x7f06500b0770 legacy=0x7f0650149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 1 monclient: found mon.noname-b 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06500ac9d0 con 0x7f06500ac5a0 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f064c001630 con 0x7f06500ac5a0 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 10 monclient: dump: 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:14.359 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f064c001880 con 0x7f06500ac5a0 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 1 -- 172.21.15.155:0/2745296485 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f064c001c10 con 0x7f06500ac5a0 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f0647fff640 10 monclient: dump: 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:14.360 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f065773e640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f065773e640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.352+0000 7f065773e640 4 set_mon_vals no callback set 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.353+0000 7f065773e640 10 monclient: shutdown 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.353+0000 7f065773e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.353+0000 7f065773e640 1 -- 172.21.15.155:0/2745296485 >> v1:172.21.15.155:6790/0 conn(0x7f06500ac5a0 legacy=0x7f0650151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.353+0000 7f065773e640 1 -- 172.21.15.155:0/2745296485 shutdown_connections 2024-03-29T11:39:14.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.353+0000 7f065773e640 1 -- 172.21.15.155:0/2745296485 >> 172.21.15.155:0/2745296485 conn(0x7f065006b660 msgr2=0x7f065006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:14.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.354+0000 7f065773e640 1 -- 172.21.15.155:0/2745296485 shutdown_connections 2024-03-29T11:39:14.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.357+0000 7f065773e640 1 -- 172.21.15.155:0/2745296485 wait complete. 2024-03-29T11:39:14.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.358+0000 7f065773e640 10 monclient: build_initial_monmap 2024-03-29T11:39:14.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.358+0000 7f065773e640 10 monclient: monmap: 2024-03-29T11:39:14.362 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:14.362 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:14.362 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:39:14.362 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:39:14.363 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:14.363 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:14.363 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:14.363 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:39:14.363 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:39:14.363 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:14.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.358+0000 7f065773e640 1 librados: starting msgr at 2024-03-29T11:39:14.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.358+0000 7f065773e640 1 librados: starting objecter 2024-03-29T11:39:14.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.358+0000 7f065773e640 1 Processor -- start 2024-03-29T11:39:14.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 1 -- start start 2024-03-29T11:39:14.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 1 librados: setting wanted keys 2024-03-29T11:39:14.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 1 librados: calling monclient init 2024-03-29T11:39:14.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 10 monclient: init 2024-03-29T11:39:14.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:14.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:14.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 10 monclient(hunting): picked mon.noname-c con 0x7f06500ac5a0 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:14.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 10 monclient(hunting): picked mon.noname-b con 0x7f06500b0770 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:14.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 10 monclient(hunting): picked mon.noname-a con 0x7f065014a700 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:14.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065012a3a0 con 0x7f065014a700 2024-03-29T11:39:14.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065012a680 con 0x7f06500b0770 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0650128020 con 0x7f06500ac5a0 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.359+0000 7f065773e640 10 monclient(hunting): authenticate will time out at 1136.486818s 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.362+0000 7f06554da640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f06500ac5a0 0x7f0650128820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50368/0 (socket says 172.21.15.155:50368) 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.362+0000 7f06554da640 1 -- 172.21.15.155:0/1578910232 learned_addr learned my addr 172.21.15.155:0/1578910232 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.362+0000 7f0645ffb640 10 client.?.objecter ms_handle_connect 0x7f06500ac5a0 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.362+0000 7f0645ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.363+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007427783 0 0) 0x7f0650128020 con 0x7f06500ac5a0 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.363+0000 7f0645ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.363+0000 7f0645ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.363+0000 7f0645ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.363+0000 7f0645ffb640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:39:14.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.363+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0634004eb0 con 0x7f06500ac5a0 2024-03-29T11:39:14.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.363+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2215026947 0 0) 0x7f0634004eb0 con 0x7f06500ac5a0 2024-03-29T11:39:14.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.363+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0634005300 con 0x7f06500ac5a0 2024-03-29T11:39:14.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.364+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f06400019f0 con 0x7f06500ac5a0 2024-03-29T11:39:14.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.364+0000 7f0645ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:14.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.364+0000 7f0645ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:14.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.364+0000 7f0645ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:14.366 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:14.366 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:14.366 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:14.366 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:14.366 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:14.367 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:14.367 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:14.367 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:14.367 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:14.367 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:14.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.364+0000 7f0645ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:14.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.364+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3559398862 0 0) 0x7f0634005300 con 0x7f06500ac5a0 2024-03-29T11:39:14.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.364+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 >> v1:172.21.15.183:6789/0 conn(0x7f06500b0770 legacy=0x7f0650128f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:14.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 >> v1:172.21.15.155:6789/0 conn(0x7f065014a700 legacy=0x7f06501e5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:14.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f0645ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:14.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f0645ffb640 1 monclient: found mon.c 2024-03-29T11:39:14.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f0645ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:14.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f0645ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0650128300 con 0x7f06500ac5a0 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f0645ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f0645ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f0645ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f065773e640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f065773e640 10 monclient: _renew_subs 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f065773e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f065773e640 1 -- 172.21.15.155:0/1578910232 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06501df730 con 0x7f06500ac5a0 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f065773e640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f065773e640 10 monclient: _renew_subs 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f065773e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f065773e640 1 -- 172.21.15.155:0/1578910232 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06501deea0 con 0x7f06500ac5a0 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.365+0000 7f065773e640 1 librados: init done 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.366+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0640001cb0 con 0x7f06500ac5a0 2024-03-29T11:39:14.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.366+0000 7f0645ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.366+0000 7f06477fe640 4 set_mon_vals no callback set 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.366+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f06400019f0 con 0x7f06500ac5a0 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.366+0000 7f0645ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.366+0000 7f0645ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.366+0000 7f0645ffb640 10 monclient: dump: 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:14.370 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:14.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.370+0000 7f065773e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:14.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.370+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f0640003920 con 0x7f06500ac5a0 2024-03-29T11:39:14.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.370+0000 7f065773e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:14.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.370+0000 7f065773e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:14.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.370+0000 7f065773e640 1 -- 172.21.15.155:0/1578910232 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0630005080 con 0x7f06500ac5a0 2024-03-29T11:39:14.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.371+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f06400847d0 con 0x7f06500ac5a0 2024-03-29T11:39:14.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.371+0000 7f0645ffb640 10 client.4409.objecter ms_dispatch 0x7f0650000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:14.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.371+0000 7f0645ffb640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:14.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.371+0000 7f0645ffb640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:14.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.371+0000 7f0645ffb640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:39:14.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.371+0000 7f0654cd9640 10 monclient: get_auth_request con 0x7f063406dfc0 auth_method 0 2024-03-29T11:39:14.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.371+0000 7f0645ffb640 10 client.4409.objecter ms_handle_connect 0x7f063406dfc0 2024-03-29T11:39:14.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.380+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0640050110 con 0x7f06500ac5a0 2024-03-29T11:39:14.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.380+0000 7f0645ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:14.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.380+0000 7f0645ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:14.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.686+0000 7f065773e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:14.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.686+0000 7f065773e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:14.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.686+0000 7f065773e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:14.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.686+0000 7f065773e640 1 -- 172.21.15.155:0/1578910232 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0630005330 con 0x7f06500ac5a0 2024-03-29T11:39:14.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.687+0000 7f0645ffb640 1 -- 172.21.15.155:0/1578910232 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f0640083e80 con 0x7f06500ac5a0 2024-03-29T11:39:14.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.687+0000 7f0645ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:14.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.687+0000 7f0645ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:14.689 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:39:14.689 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 10 librados: watch_flush enter 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 10 librados: watch_flush exit 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 1 -- 172.21.15.155:0/1578910232 >> v1:172.21.15.155:6816/1760814708 conn(0x7f063406dfc0 legacy=0x7f0634070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 10 monclient: shutdown 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 1 -- 172.21.15.155:0/1578910232 >> v1:172.21.15.155:6790/0 conn(0x7f06500ac5a0 legacy=0x7f0650128820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 1 -- 172.21.15.155:0/1578910232 shutdown_connections 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 1 -- 172.21.15.155:0/1578910232 >> 172.21.15.155:0/1578910232 conn(0x7f065006b660 msgr2=0x7f0650071440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 1 -- 172.21.15.155:0/1578910232 shutdown_connections 2024-03-29T11:39:14.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 1 -- 172.21.15.155:0/1578910232 wait complete. 2024-03-29T11:39:14.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:14.690+0000 7f065773e640 1 librados: shutdown 2024-03-29T11:39:14.707 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:21.513 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:21.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.663+0000 7ff4c18db640 10 monclient: get_monmap_and_config 2024-03-29T11:39:21.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.663+0000 7ff4c18db640 10 monclient: build_initial_monmap 2024-03-29T11:39:21.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.663+0000 7ff4c18db640 10 monclient: monmap: 2024-03-29T11:39:21.665 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:21.665 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:21.665 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:39:21.665183+0000 2024-03-29T11:39:21.665 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:39:21.665183+0000 2024-03-29T11:39:21.665 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:21.665 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:21.666 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:21.666 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:39:21.666 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:39:21.666 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.664+0000 7ff4c18db640 1 Processor -- start 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.664+0000 7ff4c18db640 1 -- start start 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.664+0000 7ff4c18db640 10 monclient: init 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.665+0000 7ff4c18db640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.665+0000 7ff4c18db640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.665+0000 7ff4c18db640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b40389e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.665+0000 7ff4c18db640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b4038dc0 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.665+0000 7ff4c18db640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b4030660 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.665+0000 7ff4c18db640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b40c54c0 con 0x7ff4b4030660 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.665+0000 7ff4c18db640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b4036430 con 0x7ff4b4038dc0 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.665+0000 7ff4c18db640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b4039840 con 0x7ff4b40389e0 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.665+0000 7ff4c18db640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.665+0000 7ff4c18db640 10 monclient(hunting): authenticate will time out at 1143.792703s 2024-03-29T11:39:21.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.666+0000 7ff4c08d9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ff4b40389e0 0x7ff4b4033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50594/0 (socket says 172.21.15.155:50594) 2024-03-29T11:39:21.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.666+0000 7ff4c08d9640 1 -- 172.21.15.155:0/3335022831 learned_addr learned my addr 172.21.15.155:0/3335022831 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:21.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.666+0000 7ff4bb7fe640 1 -- 172.21.15.155:0/3335022831 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820906626 0 0) 0x7ff4b4039840 con 0x7ff4b40389e0 2024-03-29T11:39:21.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.666+0000 7ff4bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:21.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.666+0000 7ff4bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:21.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.666+0000 7ff4bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:21.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.666+0000 7ff4bb7fe640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:39:21.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.667+0000 7ff4bb7fe640 1 -- 172.21.15.155:0/3335022831 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ac003520 con 0x7ff4b40389e0 2024-03-29T11:39:21.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.667+0000 7ff4bb7fe640 1 -- 172.21.15.155:0/3335022831 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4223635826 0 0) 0x7ff4ac003520 con 0x7ff4b40389e0 2024-03-29T11:39:21.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.667+0000 7ff4bb7fe640 1 -- 172.21.15.155:0/3335022831 >> v1:172.21.15.183:6789/0 conn(0x7ff4b4038dc0 legacy=0x7ff4b40cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:21.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.670+0000 7ff4bb7fe640 1 -- 172.21.15.155:0/3335022831 >> v1:172.21.15.155:6789/0 conn(0x7ff4b4030660 legacy=0x7ff4b40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:21.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.670+0000 7ff4bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:21.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.670+0000 7ff4bb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:39:21.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.670+0000 7ff4bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:21.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.670+0000 7ff4bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:39:21.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.670+0000 7ff4bb7fe640 1 -- 172.21.15.155:0/3335022831 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b4035690 con 0x7ff4b40389e0 2024-03-29T11:39:21.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.670+0000 7ff4bb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4bb7fe640 1 -- 172.21.15.155:0/3335022831 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff4b0001690 con 0x7ff4b40389e0 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4c18db640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4c18db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4bb7fe640 10 monclient: dump: 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:21.675 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4c18db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4bb7fe640 1 -- 172.21.15.155:0/3335022831 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4b0001920 con 0x7ff4b40389e0 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4c18db640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4c18db640 4 set_mon_vals no callback set 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4bb7fe640 1 -- 172.21.15.155:0/3335022831 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff4b0001cf0 con 0x7ff4b40389e0 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.671+0000 7ff4bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.672+0000 7ff4bb7fe640 10 monclient: dump: 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:21.676 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:21.677 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:21.677 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:21.677 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:21.677 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:21.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.672+0000 7ff4c18db640 10 monclient: shutdown 2024-03-29T11:39:21.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.672+0000 7ff4c18db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:21.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.672+0000 7ff4c18db640 1 -- 172.21.15.155:0/3335022831 >> v1:172.21.15.155:6790/0 conn(0x7ff4b40389e0 legacy=0x7ff4b4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:21.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.672+0000 7ff4c18db640 1 -- 172.21.15.155:0/3335022831 shutdown_connections 2024-03-29T11:39:21.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.672+0000 7ff4c18db640 1 -- 172.21.15.155:0/3335022831 >> 172.21.15.155:0/3335022831 conn(0x7ff4b4005c00 msgr2=0x7ff4b4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:21.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.672+0000 7ff4c18db640 1 -- 172.21.15.155:0/3335022831 shutdown_connections 2024-03-29T11:39:21.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.672+0000 7ff4c18db640 1 -- 172.21.15.155:0/3335022831 wait complete. 2024-03-29T11:39:21.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.673+0000 7ff4c18db640 10 monclient: build_initial_monmap 2024-03-29T11:39:21.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.673+0000 7ff4c18db640 10 monclient: monmap: 2024-03-29T11:39:21.679 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:21.679 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:21.679 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:39:21.679 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:39:21.679 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:21.679 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:21.679 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.674+0000 7ff4c18db640 1 librados: starting msgr at 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.674+0000 7ff4c18db640 1 librados: starting objecter 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.674+0000 7ff4c18db640 1 Processor -- start 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.674+0000 7ff4c18db640 1 -- start start 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.674+0000 7ff4c18db640 1 librados: setting wanted keys 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.674+0000 7ff4c18db640 1 librados: calling monclient init 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.674+0000 7ff4c18db640 10 monclient: init 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.675+0000 7ff4c18db640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.676+0000 7ff4c18db640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.676+0000 7ff4c18db640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b4030660 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.676+0000 7ff4c08d9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ff4b4030660 0x7ff4b402d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50978/0 (socket says 172.21.15.155:50978) 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.676+0000 7ff4c08d9640 1 -- 172.21.15.155:0/1028066740 learned_addr learned my addr 172.21.15.155:0/1028066740 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:21.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.676+0000 7ff4c18db640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b40389e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:21.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.676+0000 7ff4b97fa640 10 client.?.objecter ms_handle_connect 0x7ff4b4030660 2024-03-29T11:39:21.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.676+0000 7ff4b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:21.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.680+0000 7ff4c18db640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b40c7890 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:21.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.680+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b415d430 con 0x7ff4b40c7890 2024-03-29T11:39:21.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.680+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b40a5780 con 0x7ff4b40389e0 2024-03-29T11:39:21.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.680+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b40c2ea0 con 0x7ff4b4030660 2024-03-29T11:39:21.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.680+0000 7ff4c18db640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:21.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4c18db640 10 monclient(hunting): authenticate will time out at 1143.807878s 2024-03-29T11:39:21.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 client.?.objecter ms_handle_connect 0x7ff4b40c7890 2024-03-29T11:39:21.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:21.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851643210 0 0) 0x7ff4b40c2ea0 con 0x7ff4b4030660 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a0004eb0 con 0x7ff4b4030660 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 client.?.objecter ms_handle_connect 0x7ff4b40389e0 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186032573 0 0) 0x7ff4b40a5780 con 0x7ff4b40389e0 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.681+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b40c2ea0 con 0x7ff4b40389e0 2024-03-29T11:39:21.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97628677 0 0) 0x7ff4b415d430 con 0x7ff4b40c7890 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b40a5780 con 0x7ff4b40c7890 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112834006 0 0) 0x7ff4b40c2ea0 con 0x7ff4b40389e0 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4b415d430 con 0x7ff4b40389e0 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff4a8001790 con 0x7ff4b40389e0 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 10 monclient(hunting): dump: 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:21.687 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3326283633 0 0) 0x7ff4b40a5780 con 0x7ff4b40c7890 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4b40c2ea0 con 0x7ff4b40c7890 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff4bc04fdd0 con 0x7ff4b40c7890 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.682+0000 7ff4b97fa640 10 monclient(hunting): dump: 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:21.688 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.683+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1540500663 0 0) 0x7ff4b40c2ea0 con 0x7ff4b40c7890 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.683+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 >> v1:172.21.15.155:6790/0 conn(0x7ff4b4030660 legacy=0x7ff4b402d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.683+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 >> v1:172.21.15.183:6789/0 conn(0x7ff4b40389e0 legacy=0x7ff4b402d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.683+0000 7ff4b97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.683+0000 7ff4b97fa640 1 monclient: found mon.a 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.683+0000 7ff4b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.683+0000 7ff4b97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.683+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b40c3030 con 0x7ff4b40c7890 2024-03-29T11:39:21.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.683+0000 7ff4b97fa640 10 monclient: _finish_auth 0 2024-03-29T11:39:21.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.683+0000 7ff4b97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:39:21.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.683+0000 7ff4b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:21.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.684+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4bc04f400 con 0x7ff4b40c7890 2024-03-29T11:39:21.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.684+0000 7ff4b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:21.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.684+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff4bc075710 con 0x7ff4b40c7890 2024-03-29T11:39:21.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.684+0000 7ff4b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:21.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.684+0000 7ff4b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:21.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.684+0000 7ff4b97fa640 10 monclient: dump: 2024-03-29T11:39:21.691 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:21.691 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:21.691 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:21.691 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.684+0000 7ff4ba7fc640 4 set_mon_vals no callback set 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.685+0000 7ff4c18db640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.685+0000 7ff4c18db640 10 monclient: _renew_subs 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.685+0000 7ff4c18db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.685+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b415cc70 con 0x7ff4b40c7890 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.685+0000 7ff4c18db640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.685+0000 7ff4c18db640 10 monclient: _renew_subs 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.685+0000 7ff4c18db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.685+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b415d100 con 0x7ff4b40c7890 2024-03-29T11:39:21.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.685+0000 7ff4c18db640 1 librados: init done 2024-03-29T11:39:21.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.687+0000 7ff4c18db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:21.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.687+0000 7ff4c18db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:21.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.687+0000 7ff4c18db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:39:21.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.687+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b4007b90 con 0x7ff4b40c7890 2024-03-29T11:39:21.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.689+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7ff4bc067bf0 con 0x7ff4b40c7890 2024-03-29T11:39:21.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.690+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7ff4bc0fe100 con 0x7ff4b40c7890 2024-03-29T11:39:21.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.690+0000 7ff4b97fa640 10 client.4407.objecter ms_dispatch 0x7ff4bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:21.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.690+0000 7ff4b97fa640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:21.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.690+0000 7ff4b97fa640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:21.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.690+0000 7ff4b97fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-29T11:39:21.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.690+0000 7ff4c08d9640 10 monclient: get_auth_request con 0x7ff4a006dbb0 auth_method 0 2024-03-29T11:39:21.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.691+0000 7ff4b97fa640 10 client.4407.objecter ms_handle_connect 0x7ff4a006dbb0 2024-03-29T11:39:21.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.696+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4bc0c99e0 con 0x7ff4b40c7890 2024-03-29T11:39:21.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.696+0000 7ff4b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:21.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:21.696+0000 7ff4b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:22.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.035+0000 7ff4c18db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:22.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.035+0000 7ff4c18db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:22.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.035+0000 7ff4c18db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:39:22.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.035+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4b4005120 con 0x7ff4b40c7890 2024-03-29T11:39:22.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.036+0000 7ff4b97fa640 1 -- 172.21.15.155:0/1028066740 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7ff4bc0cb910 con 0x7ff4b40c7890 2024-03-29T11:39:22.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.036+0000 7ff4b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:22.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.036+0000 7ff4b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:22.040 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:39:22.041 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:22.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.039+0000 7ff4c18db640 10 librados: watch_flush enter 2024-03-29T11:39:22.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.039+0000 7ff4c18db640 10 librados: watch_flush exit 2024-03-29T11:39:22.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.039+0000 7ff4c18db640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-29T11:39:22.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.039+0000 7ff4c18db640 10 client.4407.objecter successfully canceled tick 2024-03-29T11:39:22.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.040+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 >> v1:172.21.15.155:6816/1760814708 conn(0x7ff4a006dbb0 legacy=0x7ff4a0070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:22.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.040+0000 7ff4c18db640 10 monclient: shutdown 2024-03-29T11:39:22.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.040+0000 7ff4c18db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:22.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.040+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 >> v1:172.21.15.155:6789/0 conn(0x7ff4b40c7890 legacy=0x7ff4b4167350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:22.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.041+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 shutdown_connections 2024-03-29T11:39:22.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.041+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 >> 172.21.15.155:0/1028066740 conn(0x7ff4b4005c00 msgr2=0x7ff4b4034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:22.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.042+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 shutdown_connections 2024-03-29T11:39:22.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.042+0000 7ff4c18db640 1 -- 172.21.15.155:0/1028066740 wait complete. 2024-03-29T11:39:22.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:22.042+0000 7ff4c18db640 1 librados: shutdown 2024-03-29T11:39:22.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:28.868 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:29.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.015+0000 7f056404a640 10 monclient: get_monmap_and_config 2024-03-29T11:39:29.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.015+0000 7f056404a640 10 monclient: build_initial_monmap 2024-03-29T11:39:29.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.015+0000 7f056404a640 10 monclient: monmap: 2024-03-29T11:39:29.019 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:29.019 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:29.020 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:39:29.017170+0000 2024-03-29T11:39:29.020 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:39:29.017170+0000 2024-03-29T11:39:29.020 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:29.020 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:29.020 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:29.020 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:39:29.020 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:39:29.020 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:29.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.017+0000 7f056404a640 1 Processor -- start 2024-03-29T11:39:29.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.017+0000 7f056404a640 1 -- start start 2024-03-29T11:39:29.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.017+0000 7f056404a640 10 monclient: init 2024-03-29T11:39:29.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.018+0000 7f056404a640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:29.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.018+0000 7f056404a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:29.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.018+0000 7f056404a640 10 monclient(hunting): picked mon.noname-c con 0x7f055c1422b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:29.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.018+0000 7f056404a640 10 monclient(hunting): picked mon.noname-a con 0x7f055c142690 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:29.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.019+0000 7f056404a640 10 monclient(hunting): picked mon.noname-b con 0x7f055c0b17d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:29.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.019+0000 7f05625e7640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f055c0b17d0 0x7f055c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51180/0 (socket says 172.21.15.155:51180) 2024-03-29T11:39:29.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.019+0000 7f05625e7640 1 -- 172.21.15.155:0/688462152 learned_addr learned my addr 172.21.15.155:0/688462152 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:29.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.019+0000 7f056404a640 1 -- 172.21.15.155:0/688462152 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055c142c80 con 0x7f055c142690 2024-03-29T11:39:29.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.019+0000 7f056404a640 1 -- 172.21.15.155:0/688462152 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055c0aae40 con 0x7f055c1422b0 2024-03-29T11:39:29.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.019+0000 7f056404a640 1 -- 172.21.15.155:0/688462152 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055c13fbd0 con 0x7f055c0b17d0 2024-03-29T11:39:29.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.019+0000 7f056404a640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:29.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.019+0000 7f056404a640 10 monclient(hunting): authenticate will time out at 1151.146834s 2024-03-29T11:39:29.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.021+0000 7f0560de4640 1 -- 172.21.15.155:0/688462152 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193684351 0 0) 0x7f055c13fbd0 con 0x7f055c0b17d0 2024-03-29T11:39:29.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.021+0000 7f0560de4640 10 monclient(hunting): _negotiate 2024-03-29T11:39:29.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.021+0000 7f0560de4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:29.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.021+0000 7f0560de4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:29.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.021+0000 7f0560de4640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:39:29.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.021+0000 7f0560de4640 1 -- 172.21.15.155:0/688462152 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0548003520 con 0x7f055c0b17d0 2024-03-29T11:39:29.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.022+0000 7f0560de4640 1 -- 172.21.15.155:0/688462152 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 78594320 0 0) 0x7f0548003520 con 0x7f055c0b17d0 2024-03-29T11:39:29.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.022+0000 7f0560de4640 1 -- 172.21.15.155:0/688462152 >> v1:172.21.15.183:6789/0 conn(0x7f055c1422b0 legacy=0x7f055c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:29.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.022+0000 7f0560de4640 1 -- 172.21.15.155:0/688462152 >> v1:172.21.15.155:6789/0 conn(0x7f055c142690 legacy=0x7f055c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:29.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 10 monclient: _finish_hunting 0 2024-03-29T11:39:29.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 1 monclient: found mon.noname-b 2024-03-29T11:39:29.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 1 -- 172.21.15.155:0/688462152 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055c151940 con 0x7f055c0b17d0 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 10 monclient: _finish_auth 0 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 10 monclient: _check_auth_tickets 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 1 -- 172.21.15.155:0/688462152 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f054c001690 con 0x7f055c0b17d0 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f056404a640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f056404a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.025+0000 7f0560de4640 10 monclient: dump: 2024-03-29T11:39:29.030 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.026+0000 7f056404a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.026+0000 7f0560de4640 1 -- 172.21.15.155:0/688462152 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f054c001930 con 0x7f055c0b17d0 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.026+0000 7f0560de4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.026+0000 7f0560de4640 1 -- 172.21.15.155:0/688462152 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f054c001cc0 con 0x7f055c0b17d0 2024-03-29T11:39:29.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.026+0000 7f056404a640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:29.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.026+0000 7f056404a640 4 set_mon_vals no callback set 2024-03-29T11:39:29.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.026+0000 7f0560de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:29.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.026+0000 7f0560de4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:29.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.026+0000 7f0560de4640 10 monclient: dump: 2024-03-29T11:39:29.032 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:29.032 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:29.032 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:29.032 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:29.032 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.027+0000 7f056404a640 10 monclient: shutdown 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.027+0000 7f056404a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.027+0000 7f056404a640 1 -- 172.21.15.155:0/688462152 >> v1:172.21.15.155:6790/0 conn(0x7f055c0b17d0 legacy=0x7f055c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.027+0000 7f056404a640 1 -- 172.21.15.155:0/688462152 shutdown_connections 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.027+0000 7f056404a640 1 -- 172.21.15.155:0/688462152 >> 172.21.15.155:0/688462152 conn(0x7f055c066a00 msgr2=0x7f055c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.028+0000 7f056404a640 1 -- 172.21.15.155:0/688462152 shutdown_connections 2024-03-29T11:39:29.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.028+0000 7f056404a640 1 -- 172.21.15.155:0/688462152 wait complete. 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.029+0000 7f056404a640 10 monclient: build_initial_monmap 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.029+0000 7f056404a640 10 monclient: monmap: 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.029+0000 7f056404a640 1 librados: starting msgr at 2024-03-29T11:39:29.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.029+0000 7f056404a640 1 librados: starting objecter 2024-03-29T11:39:29.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.029+0000 7f056404a640 1 Processor -- start 2024-03-29T11:39:29.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.030+0000 7f056404a640 1 -- start start 2024-03-29T11:39:29.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.030+0000 7f056404a640 1 librados: setting wanted keys 2024-03-29T11:39:29.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.030+0000 7f056404a640 1 librados: calling monclient init 2024-03-29T11:39:29.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.030+0000 7f056404a640 10 monclient: init 2024-03-29T11:39:29.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f056404a640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:29.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f056404a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:29.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f056404a640 10 monclient(hunting): picked mon.noname-b con 0x7f055c0b17d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:29.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f056404a640 10 monclient(hunting): picked mon.noname-a con 0x7f055c1422b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:29.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f056404a640 10 monclient(hunting): picked mon.noname-c con 0x7f055c142690 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:29.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f056404a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055c0aba60 con 0x7f055c1422b0 2024-03-29T11:39:29.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f056404a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055c0ad0f0 con 0x7f055c0b17d0 2024-03-29T11:39:29.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f056404a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055c13eee0 con 0x7f055c142690 2024-03-29T11:39:29.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f056404a640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:29.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f056404a640 10 monclient(hunting): authenticate will time out at 1151.160598s 2024-03-29T11:39:29.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f05625e7640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f055c142690 0x7f055c1e2d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:52156/0 (socket says 172.21.15.155:52156) 2024-03-29T11:39:29.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.033+0000 7f05625e7640 1 -- 172.21.15.155:0/3585944000 learned_addr learned my addr 172.21.15.155:0/3585944000 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:29.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.036+0000 7f05527fc640 10 client.?.objecter ms_handle_connect 0x7f055c142690 2024-03-29T11:39:29.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.036+0000 7f05527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:29.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 client.?.objecter ms_handle_connect 0x7f055c1422b0 2024-03-29T11:39:29.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:29.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514500465 0 0) 0x7f055c13eee0 con 0x7f055c142690 2024-03-29T11:39:29.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:29.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:29.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0540004eb0 con 0x7f055c142690 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091724371 0 0) 0x7f055c0aba60 con 0x7f055c1422b0 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f055c13eee0 con 0x7f055c1422b0 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 client.?.objecter ms_handle_connect 0x7f055c0b17d0 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.037+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3560276505 0 0) 0x7f0540004eb0 con 0x7f055c142690 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f055c0aba60 con 0x7f055c142690 2024-03-29T11:39:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f054c001650 con 0x7f055c142690 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient(hunting): dump: 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 504246689 0 0) 0x7f055c13eee0 con 0x7f055c1422b0 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0540004eb0 con 0x7f055c1422b0 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0558001710 con 0x7f055c1422b0 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient(hunting): dump: 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:29.046 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:29.047 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:29.047 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:29.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4039454095 0 0) 0x7f055c0aba60 con 0x7f055c142690 2024-03-29T11:39:29.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 >> v1:172.21.15.183:6789/0 conn(0x7f055c0b17d0 legacy=0x7f055c1daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:29.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 >> v1:172.21.15.155:6789/0 conn(0x7f055c1422b0 legacy=0x7f055c129f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:29.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:29.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 monclient: found mon.c 2024-03-29T11:39:29.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:29.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:29.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055c13eb50 con 0x7f055c142690 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f056404a640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f056404a640 10 monclient: _renew_subs 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f056404a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f056404a640 1 -- 172.21.15.155:0/3585944000 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055c0abef0 con 0x7f055c142690 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f056404a640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f056404a640 10 monclient: _renew_subs 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f056404a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f056404a640 1 -- 172.21.15.155:0/3585944000 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f055c0ab720 con 0x7f055c142690 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f056404a640 1 librados: init done 2024-03-29T11:39:29.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f054c0017b0 con 0x7f055c142690 2024-03-29T11:39:29.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:29.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f054c004d90 con 0x7f055c142690 2024-03-29T11:39:29.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:29.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:29.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.038+0000 7f05527fc640 10 monclient: dump: 2024-03-29T11:39:29.049 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:29.049 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:29.049 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:29.049 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:29.050 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:29.050 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:29.050 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:29.050 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:29.050 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:29.050 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:29.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.039+0000 7f0553fff640 4 set_mon_vals no callback set 2024-03-29T11:39:29.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.040+0000 7f056404a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:29.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.040+0000 7f056404a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:29.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.040+0000 7f056404a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:29.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.040+0000 7f056404a640 1 -- 172.21.15.155:0/3585944000 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0544005080 con 0x7f055c142690 2024-03-29T11:39:29.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.041+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f054c004ef0 con 0x7f055c142690 2024-03-29T11:39:29.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.041+0000 7f0561de6640 10 monclient: get_auth_request con 0x7f054006de70 auth_method 0 2024-03-29T11:39:29.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.041+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f054c084c00 con 0x7f055c142690 2024-03-29T11:39:29.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.042+0000 7f05527fc640 10 client.4421.objecter ms_dispatch 0x7f055c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:29.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.042+0000 7f05527fc640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:29.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.042+0000 7f05527fc640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:29.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.042+0000 7f05527fc640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T11:39:29.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.043+0000 7f05527fc640 10 client.4421.objecter ms_handle_connect 0x7f054006de70 2024-03-29T11:39:29.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.049+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f054c050540 con 0x7f055c142690 2024-03-29T11:39:29.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.049+0000 7f05527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:29.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.049+0000 7f05527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:29.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.378+0000 7f056404a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:29.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.378+0000 7f056404a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:29.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.378+0000 7f056404a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:29.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.378+0000 7f056404a640 1 -- 172.21.15.155:0/3585944000 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0544005330 con 0x7f055c142690 2024-03-29T11:39:29.382 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:39:29.382 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:29.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.380+0000 7f05527fc640 1 -- 172.21.15.155:0/3585944000 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f054c0842b0 con 0x7f055c142690 2024-03-29T11:39:29.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.380+0000 7f05527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:29.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.380+0000 7f05527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:29.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.383+0000 7f056404a640 10 librados: watch_flush enter 2024-03-29T11:39:29.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.386+0000 7f056404a640 10 librados: watch_flush exit 2024-03-29T11:39:29.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.386+0000 7f056404a640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T11:39:29.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.386+0000 7f056404a640 10 client.4421.objecter successfully canceled tick 2024-03-29T11:39:29.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.386+0000 7f056404a640 1 -- 172.21.15.155:0/3585944000 >> v1:172.21.15.155:6816/1760814708 conn(0x7f054006de70 legacy=0x7f0540070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:29.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.386+0000 7f056404a640 10 monclient: shutdown 2024-03-29T11:39:29.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.386+0000 7f056404a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:29.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.386+0000 7f056404a640 1 -- 172.21.15.155:0/3585944000 >> v1:172.21.15.155:6790/0 conn(0x7f055c142690 legacy=0x7f055c1e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:29.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.386+0000 7f056404a640 1 -- 172.21.15.155:0/3585944000 shutdown_connections 2024-03-29T11:39:29.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.386+0000 7f056404a640 1 -- 172.21.15.155:0/3585944000 >> 172.21.15.155:0/3585944000 conn(0x7f055c066a00 msgr2=0x7f055c13c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:29.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.386+0000 7f056404a640 1 -- 172.21.15.155:0/3585944000 shutdown_connections 2024-03-29T11:39:29.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.387+0000 7f056404a640 1 -- 172.21.15.155:0/3585944000 wait complete. 2024-03-29T11:39:29.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:29.387+0000 7f056404a640 1 librados: shutdown 2024-03-29T11:39:29.406 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:36.213 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:36.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.362+0000 7fd340967640 10 monclient: get_monmap_and_config 2024-03-29T11:39:36.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.362+0000 7fd340967640 10 monclient: build_initial_monmap 2024-03-29T11:39:36.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.362+0000 7fd340967640 10 monclient: monmap: 2024-03-29T11:39:36.364 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:36.364 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:36.364 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:39:36.363906+0000 2024-03-29T11:39:36.364 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:39:36.363906+0000 2024-03-29T11:39:36.365 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:36.365 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:36.365 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:36.365 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:39:36.365 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:39:36.365 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:36.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.363+0000 7fd340967640 1 Processor -- start 2024-03-29T11:39:36.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.363+0000 7fd340967640 1 -- start start 2024-03-29T11:39:36.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.363+0000 7fd340967640 10 monclient: init 2024-03-29T11:39:36.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.364+0000 7fd340967640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:36.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.364+0000 7fd340967640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:36.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.364+0000 7fd340967640 10 monclient(hunting): picked mon.noname-b con 0x7fd33c0b2800 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:36.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.365+0000 7fd33b7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fd33c0b2800 0x7fd33c142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:40118/0 (socket says 172.21.15.155:40118) 2024-03-29T11:39:36.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.365+0000 7fd33b7fe640 1 -- 172.21.15.155:0/4228069920 learned_addr learned my addr 172.21.15.155:0/4228069920 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:36.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.365+0000 7fd340967640 10 monclient(hunting): picked mon.noname-c con 0x7fd33c0a9c00 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:36.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.365+0000 7fd340967640 10 monclient(hunting): picked mon.noname-a con 0x7fd33c0b38c0 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:36.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.365+0000 7fd340967640 1 -- 172.21.15.155:0/4228069920 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c141eb0 con 0x7fd33c0b38c0 2024-03-29T11:39:36.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.365+0000 7fd340967640 1 -- 172.21.15.155:0/4228069920 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0b22d0 con 0x7fd33c0a9c00 2024-03-29T11:39:36.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.365+0000 7fd340967640 1 -- 172.21.15.155:0/4228069920 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c142d30 con 0x7fd33c0b2800 2024-03-29T11:39:36.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.365+0000 7fd340967640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:36.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.365+0000 7fd340967640 10 monclient(hunting): authenticate will time out at 1158.492462s 2024-03-29T11:39:36.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.368+0000 7fd33a7fc640 1 -- 172.21.15.155:0/4228069920 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624725755 0 0) 0x7fd33c142d30 con 0x7fd33c0b2800 2024-03-29T11:39:36.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.368+0000 7fd33a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:36.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.368+0000 7fd33a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:36.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.368+0000 7fd33a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:36.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.368+0000 7fd33a7fc640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:39:36.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.368+0000 7fd33a7fc640 1 -- 172.21.15.155:0/4228069920 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd324003520 con 0x7fd33c0b2800 2024-03-29T11:39:36.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.368+0000 7fd33a7fc640 1 -- 172.21.15.155:0/4228069920 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1343629540 0 0) 0x7fd324003520 con 0x7fd33c0b2800 2024-03-29T11:39:36.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.368+0000 7fd33a7fc640 1 -- 172.21.15.155:0/4228069920 >> v1:172.21.15.183:6789/0 conn(0x7fd33c0a9c00 legacy=0x7fd33c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:36.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.368+0000 7fd33a7fc640 1 -- 172.21.15.155:0/4228069920 >> v1:172.21.15.155:6789/0 conn(0x7fd33c0b38c0 legacy=0x7fd33c14d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:36.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:36.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:39:36.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:36.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:39:36.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 1 -- 172.21.15.155:0/4228069920 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0aaf90 con 0x7fd33c0b2800 2024-03-29T11:39:36.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:36.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:36.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:36.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 1 -- 172.21.15.155:0/4228069920 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd330001690 con 0x7fd33c0b2800 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 10 monclient: dump: 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 1 -- 172.21.15.155:0/4228069920 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3300017f0 con 0x7fd33c0b2800 2024-03-29T11:39:36.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.370+0000 7fd33a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.371+0000 7fd33a7fc640 1 -- 172.21.15.155:0/4228069920 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd330001c40 con 0x7fd33c0b2800 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.371+0000 7fd33a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.371+0000 7fd33a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.371+0000 7fd33a7fc640 10 monclient: dump: 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:36.376 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:36.377 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:36.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.371+0000 7fd340967640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:39:36.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.371+0000 7fd340967640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:36.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.371+0000 7fd340967640 4 set_mon_vals no callback set 2024-03-29T11:39:36.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.372+0000 7fd340967640 10 monclient: shutdown 2024-03-29T11:39:36.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.372+0000 7fd340967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:36.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.372+0000 7fd340967640 1 -- 172.21.15.155:0/4228069920 >> v1:172.21.15.155:6790/0 conn(0x7fd33c0b2800 legacy=0x7fd33c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:36.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.374+0000 7fd340967640 1 -- 172.21.15.155:0/4228069920 shutdown_connections 2024-03-29T11:39:36.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.374+0000 7fd340967640 1 -- 172.21.15.155:0/4228069920 >> 172.21.15.155:0/4228069920 conn(0x7fd33c066a00 msgr2=0x7fd33c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:36.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.375+0000 7fd340967640 1 -- 172.21.15.155:0/4228069920 shutdown_connections 2024-03-29T11:39:36.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.375+0000 7fd340967640 1 -- 172.21.15.155:0/4228069920 wait complete. 2024-03-29T11:39:36.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.377+0000 7fd340967640 10 monclient: build_initial_monmap 2024-03-29T11:39:36.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.377+0000 7fd340967640 10 monclient: monmap: 2024-03-29T11:39:36.379 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:36.379 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:36.379 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:39:36.379 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:39:36.379 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:36.379 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:36.379 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:36.379 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:39:36.379 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:39:36.379 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:36.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.377+0000 7fd340967640 1 librados: starting msgr at 2024-03-29T11:39:36.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.377+0000 7fd340967640 1 librados: starting objecter 2024-03-29T11:39:36.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.377+0000 7fd340967640 1 Processor -- start 2024-03-29T11:39:36.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.378+0000 7fd340967640 1 -- start start 2024-03-29T11:39:36.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.378+0000 7fd340967640 1 librados: setting wanted keys 2024-03-29T11:39:36.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.378+0000 7fd340967640 1 librados: calling monclient init 2024-03-29T11:39:36.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.378+0000 7fd340967640 10 monclient: init 2024-03-29T11:39:36.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.379+0000 7fd340967640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:36.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.379+0000 7fd340967640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:36.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.380+0000 7fd340967640 10 monclient(hunting): picked mon.noname-b con 0x7fd33c0a9c00 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:36.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.382+0000 7fd340967640 10 monclient(hunting): picked mon.noname-a con 0x7fd33c0b2800 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:36.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.382+0000 7fd33b7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd33c0a9c00 0x7fd33c0ae280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:55569/0 (socket says 172.21.15.155:55569) 2024-03-29T11:39:36.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.382+0000 7fd33b7fe640 1 -- 172.21.15.155:0/1783804133 learned_addr learned my addr 172.21.15.155:0/1783804133 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:36.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.382+0000 7fd323fff640 10 client.?.objecter ms_handle_connect 0x7fd33c0a9c00 2024-03-29T11:39:36.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.383+0000 7fd323fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:36.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.382+0000 7fd340967640 10 monclient(hunting): picked mon.noname-c con 0x7fd33c0b38c0 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:36.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.383+0000 7fd323fff640 10 client.?.objecter ms_handle_connect 0x7fd33c0b38c0 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.383+0000 7fd323fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.383+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c1da110 con 0x7fd33c0b2800 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.383+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0ae990 con 0x7fd33c0a9c00 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.383+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0aeb20 con 0x7fd33c0b38c0 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.383+0000 7fd340967640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.383+0000 7fd340967640 10 monclient(hunting): authenticate will time out at 1158.510495s 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.383+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364734716 0 0) 0x7fd33c0ae990 con 0x7fd33c0a9c00 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.383+0000 7fd323fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd328004eb0 con 0x7fd33c0a9c00 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873372737 0 0) 0x7fd33c0aeb20 con 0x7fd33c0b38c0 2024-03-29T11:39:36.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd33c0ae990 con 0x7fd33c0b38c0 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2470033538 0 0) 0x7fd328004eb0 con 0x7fd33c0a9c00 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd33c0aeb20 con 0x7fd33c0a9c00 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.384+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd330001b80 con 0x7fd33c0a9c00 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.385+0000 7fd323fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.385+0000 7fd323fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.385+0000 7fd323fff640 10 monclient(hunting): dump: 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:36.392 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.385+0000 7fd323fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.385+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 632269732 0 0) 0x7fd33c0ae990 con 0x7fd33c0b38c0 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.385+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd328004eb0 con 0x7fd33c0b38c0 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.385+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3282270873 0 0) 0x7fd33c0aeb20 con 0x7fd33c0a9c00 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 >> v1:172.21.15.155:6790/0 conn(0x7fd33c0b38c0 legacy=0x7fd33c1e1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:36.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 >> v1:172.21.15.155:6789/0 conn(0x7fd33c0b2800 legacy=0x7fd33c1da610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 1 monclient: found mon.b 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c1dad20 con 0x7fd33c0a9c00 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 10 client.?.objecter ms_handle_connect 0x7fd33c0b2800 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd340967640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:39:36.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.386+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd330004b00 con 0x7fd33c0a9c00 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd340967640 10 monclient: _renew_subs 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd340967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c0affd0 con 0x7fd33c0a9c00 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd323fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd330004e90 con 0x7fd33c0a9c00 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd323fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd323fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd323fff640 10 monclient: dump: 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:36.395 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:36.396 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:36.396 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:36.396 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:36.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd339ffb640 4 set_mon_vals no callback set 2024-03-29T11:39:36.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd340967640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:36.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd340967640 10 monclient: _renew_subs 2024-03-29T11:39:36.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd340967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:39:36.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c0adbf0 con 0x7fd33c0a9c00 2024-03-29T11:39:36.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.387+0000 7fd340967640 1 librados: init done 2024-03-29T11:39:36.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.390+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fd33001ae60 con 0x7fd33c0a9c00 2024-03-29T11:39:36.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.391+0000 7fd340967640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:36.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.391+0000 7fd340967640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:36.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.391+0000 7fd340967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:39:36.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.391+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c1da2a0 con 0x7fd33c0a9c00 2024-03-29T11:39:36.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.391+0000 7fd33affd640 10 monclient: get_auth_request con 0x7fd32806dd50 auth_method 0 2024-03-29T11:39:36.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.392+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fd330084b70 con 0x7fd33c0a9c00 2024-03-29T11:39:36.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.392+0000 7fd323fff640 10 client.4399.objecter ms_dispatch 0x7fd33c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:36.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.392+0000 7fd323fff640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:36.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.392+0000 7fd323fff640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:36.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.392+0000 7fd323fff640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-29T11:39:36.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.392+0000 7fd323fff640 10 client.4399.objecter ms_handle_connect 0x7fd32806dd50 2024-03-29T11:39:36.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.401+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd330084e60 con 0x7fd33c0a9c00 2024-03-29T11:39:36.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.401+0000 7fd323fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:36.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.401+0000 7fd323fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:36.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.728+0000 7fd340967640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:36.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.728+0000 7fd340967640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:36.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.728+0000 7fd340967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:39:36.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.728+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd33c04f730 con 0x7fd33c0a9c00 2024-03-29T11:39:36.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.730+0000 7fd323fff640 1 -- 172.21.15.155:0/1783804133 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fd330050450 con 0x7fd33c0a9c00 2024-03-29T11:39:36.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.730+0000 7fd323fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:36.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.730+0000 7fd323fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:36.732 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:39:36.732 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:36.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.733+0000 7fd340967640 10 librados: watch_flush enter 2024-03-29T11:39:36.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.734+0000 7fd340967640 10 librados: watch_flush exit 2024-03-29T11:39:36.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.734+0000 7fd340967640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-29T11:39:36.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.734+0000 7fd340967640 10 client.4399.objecter successfully canceled tick 2024-03-29T11:39:36.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.734+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 >> v1:172.21.15.155:6816/1760814708 conn(0x7fd32806dd50 legacy=0x7fd328070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:36.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.734+0000 7fd340967640 10 monclient: shutdown 2024-03-29T11:39:36.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.734+0000 7fd340967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:36.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.734+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 >> v1:172.21.15.183:6789/0 conn(0x7fd33c0a9c00 legacy=0x7fd33c0ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:36.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.734+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 shutdown_connections 2024-03-29T11:39:36.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.734+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 >> 172.21.15.155:0/1783804133 conn(0x7fd33c066a00 msgr2=0x7fd33c145240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:36.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.734+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 shutdown_connections 2024-03-29T11:39:36.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.735+0000 7fd340967640 1 -- 172.21.15.155:0/1783804133 wait complete. 2024-03-29T11:39:36.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:36.735+0000 7fd340967640 1 librados: shutdown 2024-03-29T11:39:36.753 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:43.560 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:43.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.705+0000 7fbefc2ff640 10 monclient: get_monmap_and_config 2024-03-29T11:39:43.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.705+0000 7fbefc2ff640 10 monclient: build_initial_monmap 2024-03-29T11:39:43.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.705+0000 7fbefc2ff640 10 monclient: monmap: 2024-03-29T11:39:43.706 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:43.706 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:43.706 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:39:43.705678+0000 2024-03-29T11:39:43.706 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:39:43.705678+0000 2024-03-29T11:39:43.706 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:43.706 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:43.707 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:43.707 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:39:43.707 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:39:43.707 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:43.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.706+0000 7fbefc2ff640 1 Processor -- start 2024-03-29T11:39:43.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.706+0000 7fbefc2ff640 1 -- start start 2024-03-29T11:39:43.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.706+0000 7fbefc2ff640 10 monclient: init 2024-03-29T11:39:43.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.707+0000 7fbefc2ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:43.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.707+0000 7fbefc2ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:43.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.710+0000 7fbefc2ff640 10 monclient(hunting): picked mon.noname-c con 0x7fbef4143360 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:43.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.710+0000 7fbefc2ff640 10 monclient(hunting): picked mon.noname-b con 0x7fbef4143da0 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:43.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.710+0000 7fbefc2ff640 10 monclient(hunting): picked mon.noname-a con 0x7fbef4072f70 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:43.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.710+0000 7fbefc2ff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef406db30 con 0x7fbef4072f70 2024-03-29T11:39:43.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.710+0000 7fbefc2ff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef4143740 con 0x7fbef4143360 2024-03-29T11:39:43.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.710+0000 7fbefc2ff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef41438d0 con 0x7fbef4143da0 2024-03-29T11:39:43.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.710+0000 7fbefc2ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:43.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.710+0000 7fbefc2ff640 10 monclient(hunting): authenticate will time out at 1165.836135s 2024-03-29T11:39:43.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.710+0000 7fbef989a640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fbef4143da0 0x7fbef4072860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55286/0 (socket says 172.21.15.155:55286) 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.710+0000 7fbef989a640 1 -- 172.21.15.155:0/4077600502 learned_addr learned my addr 172.21.15.155:0/4077600502 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 1 -- 172.21.15.155:0/4077600502 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729845036 0 0) 0x7fbef41438d0 con 0x7fbef4143da0 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 10 monclient(hunting): _negotiate 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 1 -- 172.21.15.155:0/4077600502 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee0003520 con 0x7fbef4143da0 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 1 -- 172.21.15.155:0/4077600502 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073480061 0 0) 0x7fbef4143740 con 0x7fbef4143360 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 10 monclient(hunting): _negotiate 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.711+0000 7fbef9099640 1 -- 172.21.15.155:0/4077600502 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef41438d0 con 0x7fbef4143360 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.712+0000 7fbef9099640 1 -- 172.21.15.155:0/4077600502 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2025285759 0 0) 0x7fbef41438d0 con 0x7fbef4143360 2024-03-29T11:39:43.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.712+0000 7fbef9099640 1 -- 172.21.15.155:0/4077600502 >> v1:172.21.15.155:6790/0 conn(0x7fbef4143da0 legacy=0x7fbef4072860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.712+0000 7fbef9099640 1 -- 172.21.15.155:0/4077600502 >> v1:172.21.15.155:6789/0 conn(0x7fbef4072f70 legacy=0x7fbef414b490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:43.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 10 monclient: _finish_hunting 0 2024-03-29T11:39:43.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 1 monclient: found mon.noname-c 2024-03-29T11:39:43.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:43.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:39:43.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 1 -- 172.21.15.155:0/4077600502 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef40aec90 con 0x7fbef4143360 2024-03-29T11:39:43.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 10 monclient: _finish_auth 0 2024-03-29T11:39:43.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 10 monclient: _check_auth_tickets 2024-03-29T11:39:43.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 1 -- 172.21.15.155:0/4077600502 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fbef00015a0 con 0x7fbef4143360 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbef9099640 10 monclient: dump: 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:43.719 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbefc2ff640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.714+0000 7fbefc2ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.715+0000 7fbef9099640 1 -- 172.21.15.155:0/4077600502 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbef00017c0 con 0x7fbef4143360 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.715+0000 7fbef9099640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.715+0000 7fbef9099640 1 -- 172.21.15.155:0/4077600502 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fbef0001b50 con 0x7fbef4143360 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.715+0000 7fbefc2ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.715+0000 7fbefc2ff640 4 set_mon_vals no callback set 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.715+0000 7fbef9099640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.715+0000 7fbef9099640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.715+0000 7fbef9099640 10 monclient: dump: 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:43.720 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:43.721 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:43.721 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:43.721 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:43.721 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:43.721 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:43.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.716+0000 7fbefc2ff640 10 monclient: shutdown 2024-03-29T11:39:43.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.716+0000 7fbefc2ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:43.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.716+0000 7fbefc2ff640 1 -- 172.21.15.155:0/4077600502 >> v1:172.21.15.183:6789/0 conn(0x7fbef4143360 legacy=0x7fbef40ae580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.716+0000 7fbefc2ff640 1 -- 172.21.15.155:0/4077600502 shutdown_connections 2024-03-29T11:39:43.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.716+0000 7fbefc2ff640 1 -- 172.21.15.155:0/4077600502 >> 172.21.15.155:0/4077600502 conn(0x7fbef40b3c80 msgr2=0x7fbef40b4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:43.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.716+0000 7fbefc2ff640 1 -- 172.21.15.155:0/4077600502 shutdown_connections 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.717+0000 7fbefc2ff640 1 -- 172.21.15.155:0/4077600502 wait complete. 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.718+0000 7fbefc2ff640 10 monclient: build_initial_monmap 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.718+0000 7fbefc2ff640 10 monclient: monmap: 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:39:43.724 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.718+0000 7fbefc2ff640 1 librados: starting msgr at 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.718+0000 7fbefc2ff640 1 librados: starting objecter 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.718+0000 7fbefc2ff640 1 Processor -- start 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.718+0000 7fbefc2ff640 1 -- start start 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.718+0000 7fbefc2ff640 1 librados: setting wanted keys 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.718+0000 7fbefc2ff640 1 librados: calling monclient init 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.718+0000 7fbefc2ff640 10 monclient: init 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.719+0000 7fbefc2ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.719+0000 7fbefc2ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.719+0000 7fbefc2ff640 10 monclient(hunting): picked mon.noname-a con 0x7fbef4072f70 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.721+0000 7fbefc2ff640 10 monclient(hunting): picked mon.noname-c con 0x7fbef4143da0 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.721+0000 7fbefc2ff640 10 monclient(hunting): picked mon.noname-b con 0x7fbef40aac20 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.721+0000 7fbefc2ff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef411c190 con 0x7fbef4072f70 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.721+0000 7fbefc2ff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef40ab190 con 0x7fbef40aac20 2024-03-29T11:39:43.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.722+0000 7fbefc2ff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef40a9cc0 con 0x7fbef4143da0 2024-03-29T11:39:43.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.722+0000 7fbefc2ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:43.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.722+0000 7fbefc2ff640 10 monclient(hunting): authenticate will time out at 1165.848289s 2024-03-29T11:39:43.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.722+0000 7fbef989a640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fbef4143da0 0x7fbef411eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:48002/0 (socket says 172.21.15.155:48002) 2024-03-29T11:39:43.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.722+0000 7fbef989a640 1 -- 172.21.15.155:0/1781387651 learned_addr learned my addr 172.21.15.155:0/1781387651 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:43.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.724+0000 7fbeeaffd640 10 client.?.objecter ms_handle_connect 0x7fbef4143da0 2024-03-29T11:39:43.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.724+0000 7fbeeaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:43.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.724+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595824268 0 0) 0x7fbef40a9cc0 con 0x7fbef4143da0 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.724+0000 7fbeeaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.724+0000 7fbeeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.724+0000 7fbeeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.724+0000 7fbeeaffd640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.724+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed8004eb0 con 0x7fbef4143da0 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1307921420 0 0) 0x7fbed8004eb0 con 0x7fbef4143da0 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbef40a9cc0 con 0x7fbef4143da0 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fbee40018f0 con 0x7fbef4143da0 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 10 monclient(hunting): dump: 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:43.730 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2233728274 0 0) 0x7fbef40a9cc0 con 0x7fbef4143da0 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 >> v1:172.21.15.183:6789/0 conn(0x7fbef40aac20 legacy=0x7fbef41db780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 >> v1:172.21.15.155:6789/0 conn(0x7fbef4072f70 legacy=0x7fbef40aa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 1 monclient: found mon.c 2024-03-29T11:39:43.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.726+0000 7fbeeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbeeaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef411f1c0 con 0x7fbef4143da0 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbeeaffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbeeaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbeeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbefc2ff640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbefc2ff640 10 monclient: _renew_subs 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbefc2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbefc2ff640 1 -- 172.21.15.155:0/1781387651 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef411f470 con 0x7fbef4143da0 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbefc2ff640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbefc2ff640 10 monclient: _renew_subs 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbefc2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbefc2ff640 1 -- 172.21.15.155:0/1781387651 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef41dc070 con 0x7fbef4143da0 2024-03-29T11:39:43.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.727+0000 7fbefc2ff640 1 librados: init done 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.730+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbee40047e0 con 0x7fbef4143da0 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.730+0000 7fbeeaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.730+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fbee4004c30 con 0x7fbef4143da0 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.730+0000 7fbeeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.730+0000 7fbeeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.730+0000 7fbeeaffd640 10 monclient: dump: 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:43.734 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:43.735 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:43.735 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:43.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.730+0000 7fbef8898640 4 set_mon_vals no callback set 2024-03-29T11:39:43.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.731+0000 7fbefc2ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:43.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.731+0000 7fbefc2ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:43.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.731+0000 7fbefc2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:43.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.731+0000 7fbefc2ff640 1 -- 172.21.15.155:0/1781387651 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbedc005080 con 0x7fbef4143da0 2024-03-29T11:39:43.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.732+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fbee4004d90 con 0x7fbef4143da0 2024-03-29T11:39:43.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.733+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fbee4084b30 con 0x7fbef4143da0 2024-03-29T11:39:43.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.733+0000 7fbeeaffd640 10 client.4433.objecter ms_dispatch 0x7fbef4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:43.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.733+0000 7fbeeaffd640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:43.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.733+0000 7fbeeaffd640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:43.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.733+0000 7fbeeaffd640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-29T11:39:43.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.734+0000 7fbefa09b640 10 monclient: get_auth_request con 0x7fbed806dd00 auth_method 0 2024-03-29T11:39:43.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.734+0000 7fbeeaffd640 10 client.4433.objecter ms_handle_connect 0x7fbed806dd00 2024-03-29T11:39:43.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.740+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbee4050410 con 0x7fbef4143da0 2024-03-29T11:39:43.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.740+0000 7fbeeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:43.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:43.740+0000 7fbeeaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:44.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.067+0000 7fbefc2ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:44.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.067+0000 7fbefc2ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:44.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.067+0000 7fbefc2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:44.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.067+0000 7fbefc2ff640 1 -- 172.21.15.155:0/1781387651 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbedc005330 con 0x7fbef4143da0 2024-03-29T11:39:44.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.070+0000 7fbeeaffd640 1 -- 172.21.15.155:0/1781387651 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fbee4052340 con 0x7fbef4143da0 2024-03-29T11:39:44.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.070+0000 7fbeeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:44.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.070+0000 7fbeeaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:44.072 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:39:44.072 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:44.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.074+0000 7fbefc2ff640 10 librados: watch_flush enter 2024-03-29T11:39:44.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.075+0000 7fbefc2ff640 10 librados: watch_flush exit 2024-03-29T11:39:44.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.075+0000 7fbefc2ff640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-29T11:39:44.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.075+0000 7fbefc2ff640 10 client.4433.objecter successfully canceled tick 2024-03-29T11:39:44.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.076+0000 7fbefc2ff640 1 -- 172.21.15.155:0/1781387651 >> v1:172.21.15.155:6816/1760814708 conn(0x7fbed806dd00 legacy=0x7fbed80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:44.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.076+0000 7fbefc2ff640 10 monclient: shutdown 2024-03-29T11:39:44.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.076+0000 7fbefc2ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:44.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.076+0000 7fbefc2ff640 1 -- 172.21.15.155:0/1781387651 >> v1:172.21.15.155:6790/0 conn(0x7fbef4143da0 legacy=0x7fbef411eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:44.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.076+0000 7fbefc2ff640 1 -- 172.21.15.155:0/1781387651 shutdown_connections 2024-03-29T11:39:44.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.076+0000 7fbefc2ff640 1 -- 172.21.15.155:0/1781387651 >> 172.21.15.155:0/1781387651 conn(0x7fbef40b3c80 msgr2=0x7fbef40b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:44.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.076+0000 7fbefc2ff640 1 -- 172.21.15.155:0/1781387651 shutdown_connections 2024-03-29T11:39:44.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.076+0000 7fbefc2ff640 1 -- 172.21.15.155:0/1781387651 wait complete. 2024-03-29T11:39:44.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:44.076+0000 7fbefc2ff640 1 librados: shutdown 2024-03-29T11:39:44.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:50.903 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:51.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.043+0000 7f6fd533b640 10 monclient: get_monmap_and_config 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.043+0000 7f6fd533b640 10 monclient: build_initial_monmap 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.043+0000 7f6fd533b640 10 monclient: monmap: 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:39:51.044392+0000 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:39:51.044392+0000 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:39:51.046 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:51.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.045+0000 7f6fd533b640 1 Processor -- start 2024-03-29T11:39:51.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.045+0000 7f6fd533b640 1 -- start start 2024-03-29T11:39:51.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.045+0000 7f6fd533b640 10 monclient: init 2024-03-29T11:39:51.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fd533b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:51.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fd533b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:51.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fd533b640 10 monclient(hunting): picked mon.noname-a con 0x7f6fd00a9ac0 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:51.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fd533b640 10 monclient(hunting): picked mon.noname-c con 0x7f6fd013ee90 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:51.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fd533b640 10 monclient(hunting): picked mon.noname-b con 0x7f6fd00af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:51.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fd533b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00afd10 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fd533b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00af6f0 con 0x7f6fd013ee90 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fd533b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00aac80 con 0x7f6fd00af900 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fd533b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fd533b640 10 monclient(hunting): authenticate will time out at 1173.172192s 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fcf7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f6fd00af900 0x7f6fd0149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:37888/0 (socket says 172.21.15.155:37888) 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.046+0000 7f6fcf7fe640 1 -- 172.21.15.155:0/3898861891 learned_addr learned my addr 172.21.15.155:0/3898861891 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.047+0000 7f6fcdffb640 1 -- 172.21.15.155:0/3898861891 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952740345 0 0) 0x7f6fd00aac80 con 0x7f6fd00af900 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.047+0000 7f6fcdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.047+0000 7f6fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.047+0000 7f6fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.047+0000 7f6fcdffb640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.047+0000 7f6fcdffb640 1 -- 172.21.15.155:0/3898861891 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb8003520 con 0x7f6fd00af900 2024-03-29T11:39:51.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.047+0000 7f6fcdffb640 1 -- 172.21.15.155:0/3898861891 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3275001798 0 0) 0x7f6fb8003520 con 0x7f6fd00af900 2024-03-29T11:39:51.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.047+0000 7f6fcdffb640 1 -- 172.21.15.155:0/3898861891 >> v1:172.21.15.183:6789/0 conn(0x7f6fd013ee90 legacy=0x7f6fd013b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:51.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.049+0000 7f6fcdffb640 1 -- 172.21.15.155:0/3898861891 >> v1:172.21.15.155:6789/0 conn(0x7f6fd00a9ac0 legacy=0x7f6fd00adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:51.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:51.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 1 monclient: found mon.noname-b 2024-03-29T11:39:51.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:51.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 1 -- 172.21.15.155:0/3898861891 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00aaf60 con 0x7f6fd00af900 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 1 -- 172.21.15.155:0/3898861891 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6fc4001690 con 0x7f6fd00af900 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 10 monclient: dump: 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:51.057 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:51.058 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:51.058 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:51.058 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:51.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fd533b640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:39:51.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fd533b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:51.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 1 -- 172.21.15.155:0/3898861891 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fc4001920 con 0x7f6fd00af900 2024-03-29T11:39:51.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:51.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fd533b640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:51.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.055+0000 7f6fd533b640 4 set_mon_vals no callback set 2024-03-29T11:39:51.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.056+0000 7f6fd533b640 10 monclient: shutdown 2024-03-29T11:39:51.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.056+0000 7f6fd533b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:51.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.056+0000 7f6fd533b640 1 -- 172.21.15.155:0/3898861891 >> v1:172.21.15.155:6790/0 conn(0x7f6fd00af900 legacy=0x7f6fd0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:51.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.056+0000 7f6fd533b640 1 -- 172.21.15.155:0/3898861891 shutdown_connections 2024-03-29T11:39:51.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.056+0000 7f6fd533b640 1 -- 172.21.15.155:0/3898861891 >> 172.21.15.155:0/3898861891 conn(0x7f6fd00b35e0 msgr2=0x7f6fd00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:51.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.058+0000 7f6fd533b640 1 -- 172.21.15.155:0/3898861891 shutdown_connections 2024-03-29T11:39:51.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.058+0000 7f6fd533b640 1 -- 172.21.15.155:0/3898861891 wait complete. 2024-03-29T11:39:51.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.059+0000 7f6fd533b640 10 monclient: build_initial_monmap 2024-03-29T11:39:51.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.059+0000 7f6fd533b640 10 monclient: monmap: 2024-03-29T11:39:51.061 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:51.061 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:51.061 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:39:51.061 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:39:51.061 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:51.061 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:51.062 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:51.062 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:39:51.062 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:39:51.062 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:51.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.059+0000 7f6fd533b640 1 librados: starting msgr at 2024-03-29T11:39:51.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.059+0000 7f6fd533b640 1 librados: starting objecter 2024-03-29T11:39:51.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.059+0000 7f6fd533b640 1 Processor -- start 2024-03-29T11:39:51.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.059+0000 7f6fd533b640 1 -- start start 2024-03-29T11:39:51.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.059+0000 7f6fd533b640 1 librados: setting wanted keys 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.059+0000 7f6fd533b640 1 librados: calling monclient init 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.059+0000 7f6fd533b640 10 monclient: init 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.060+0000 7f6fd533b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.060+0000 7f6fd533b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.060+0000 7f6fd533b640 10 monclient(hunting): picked mon.noname-c con 0x7f6fd00a9ac0 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fd533b640 10 monclient(hunting): picked mon.noname-b con 0x7f6fd00af900 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fd533b640 10 monclient(hunting): picked mon.noname-a con 0x7f6fd013ee90 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fd533b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd0071b40 con 0x7f6fd013ee90 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fd533b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd0072550 con 0x7f6fd00af900 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fd533b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd012bd40 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fd533b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fd533b640 10 monclient(hunting): authenticate will time out at 1173.186936s 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fceffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f6fd00a9ac0 0x7f6fd013f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:41046/0 (socket says 172.21.15.155:41046) 2024-03-29T11:39:51.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fceffd640 1 -- 172.21.15.155:0/2262920060 learned_addr learned my addr 172.21.15.155:0/2262920060 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 10 client.?.objecter ms_handle_connect 0x7f6fd00a9ac0 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442972565 0 0) 0x7f6fd012bd40 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc004eb0 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 254568413 0 0) 0x7f6fbc004eb0 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fd012bd40 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6fc00018c0 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:51.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 10 monclient(hunting): dump: 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.061+0000 7f6fb77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.062+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2156189163 0 0) 0x7f6fd012bd40 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.062+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 >> v1:172.21.15.183:6789/0 conn(0x7f6fd00af900 legacy=0x7f6fd0141970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:51.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.064+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 >> v1:172.21.15.155:6789/0 conn(0x7f6fd013ee90 legacy=0x7f6fd0142080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:51.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fb77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:51.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fb77fe640 1 monclient: found mon.c 2024-03-29T11:39:51.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:51.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fb77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:51.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd012bed0 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fb77fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:51.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fb77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:51.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fc0001a20 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fcd7fa640 4 set_mon_vals no callback set 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fd533b640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fd533b640 10 monclient: _renew_subs 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fd533b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fd533b640 1 -- 172.21.15.155:0/2262920060 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd012ee80 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fd533b640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fd533b640 10 monclient: _renew_subs 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fd533b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fd533b640 1 -- 172.21.15.155:0/2262920060 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd0200d50 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.069+0000 7f6fd533b640 1 librados: init done 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.070+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6fc0004ac0 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.070+0000 7f6fb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.070+0000 7f6fb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.070+0000 7f6fb77fe640 10 monclient: dump: 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:51.072 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:51.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.071+0000 7f6fd533b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:51.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.071+0000 7f6fd533b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:51.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.071+0000 7f6fd533b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:51.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.071+0000 7f6fd533b640 1 -- 172.21.15.155:0/2262920060 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0071cf0 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.071+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f6fc001ab10 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.072+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f6fc0054970 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.072+0000 7f6fb77fe640 10 client.4439.objecter ms_dispatch 0x7f6fd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:51.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.072+0000 7f6fb77fe640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:51.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.072+0000 7f6fb77fe640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:51.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.072+0000 7f6fb77fe640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T11:39:51.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.073+0000 7f6fce7fc640 10 monclient: get_auth_request con 0x7f6fbc06dae0 auth_method 0 2024-03-29T11:39:51.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.073+0000 7f6fb77fe640 10 client.4439.objecter ms_handle_connect 0x7f6fbc06dae0 2024-03-29T11:39:51.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.081+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fc0050c10 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.081+0000 7f6fb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:51.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.081+0000 7f6fb77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:51.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.396+0000 7f6fd533b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:51.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.396+0000 7f6fd533b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:51.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.396+0000 7f6fd533b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:39:51.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.396+0000 7f6fd533b640 1 -- 172.21.15.155:0/2262920060 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fd0071cf0 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.398+0000 7f6fb77fe640 1 -- 172.21.15.155:0/2262920060 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f6fd0071cf0 con 0x7f6fd00a9ac0 2024-03-29T11:39:51.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.398+0000 7f6fb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:51.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.398+0000 7f6fb77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:51.400 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:39:51.400 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:51.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 10 librados: watch_flush enter 2024-03-29T11:39:51.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 10 librados: watch_flush exit 2024-03-29T11:39:51.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T11:39:51.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 10 client.4439.objecter successfully canceled tick 2024-03-29T11:39:51.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 1 -- 172.21.15.155:0/2262920060 >> v1:172.21.15.155:6816/1760814708 conn(0x7f6fbc06dae0 legacy=0x7f6fbc06ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:51.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 10 monclient: shutdown 2024-03-29T11:39:51.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:51.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 1 -- 172.21.15.155:0/2262920060 >> v1:172.21.15.155:6790/0 conn(0x7f6fd00a9ac0 legacy=0x7f6fd013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:51.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 1 -- 172.21.15.155:0/2262920060 shutdown_connections 2024-03-29T11:39:51.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 1 -- 172.21.15.155:0/2262920060 >> 172.21.15.155:0/2262920060 conn(0x7f6fd00b35e0 msgr2=0x7f6fd00b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:51.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 1 -- 172.21.15.155:0/2262920060 shutdown_connections 2024-03-29T11:39:51.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.402+0000 7f6fd533b640 1 -- 172.21.15.155:0/2262920060 wait complete. 2024-03-29T11:39:51.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:51.403+0000 7f6fd533b640 1 librados: shutdown 2024-03-29T11:39:51.420 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:58.226 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:58.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.393+0000 7f1355d56640 10 monclient: get_monmap_and_config 2024-03-29T11:39:58.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.393+0000 7f1355d56640 10 monclient: build_initial_monmap 2024-03-29T11:39:58.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.393+0000 7f1355d56640 10 monclient: monmap: 2024-03-29T11:39:58.395 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:58.395 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:58.395 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:39:58.394121+0000 2024-03-29T11:39:58.395 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:39:58.394121+0000 2024-03-29T11:39:58.395 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:58.395 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:58.395 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:58.395 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:39:58.395 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:39:58.395 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:58.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.395+0000 7f1355d56640 1 Processor -- start 2024-03-29T11:39:58.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.395+0000 7f1355d56640 1 -- start start 2024-03-29T11:39:58.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.395+0000 7f1355d56640 10 monclient: init 2024-03-29T11:39:58.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.396+0000 7f1355d56640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:58.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.396+0000 7f1355d56640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:58.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.396+0000 7f1355d56640 10 monclient(hunting): picked mon.noname-c con 0x7f1350071f40 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:58.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.396+0000 7f1355d56640 10 monclient(hunting): picked mon.noname-b con 0x7f135013f060 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.396+0000 7f1355d56640 10 monclient(hunting): picked mon.noname-a con 0x7f1350145ba0 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.396+0000 7f1355d56640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13500aad90 con 0x7f1350145ba0 2024-03-29T11:39:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.396+0000 7f1355d56640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13500aa260 con 0x7f1350071f40 2024-03-29T11:39:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.396+0000 7f1355d56640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13500ab000 con 0x7f135013f060 2024-03-29T11:39:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.396+0000 7f1355d56640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.396+0000 7f1355d56640 10 monclient(hunting): authenticate will time out at 1180.522816s 2024-03-29T11:39:58.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.399+0000 7f134ffff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f1350145ba0 0x7f135014d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:45156/0 (socket says 172.21.15.155:45156) 2024-03-29T11:39:58.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.399+0000 7f134ffff640 1 -- 172.21.15.155:0/593510316 learned_addr learned my addr 172.21.15.155:0/593510316 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:58.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.400+0000 7f134ffff640 1 -- 172.21.15.155:0/593510316 >> v1:172.21.15.155:6789/0 conn(0x7f1350145ba0 legacy=0x7f135014d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:39:58.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.400+0000 7f134ffff640 1 -- 172.21.15.155:0/593510316 >> v1:172.21.15.155:6789/0 conn(0x7f1350145ba0 legacy=0x7f135014d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:39:58.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.400+0000 7f134ffff640 1 --1- 172.21.15.155:0/593510316 >> v1:172.21.15.155:6789/0 conn(0x7f1350145ba0 0x7f135014d830 :-1 s=OPENED pgs=528383 cs=1 l=1).handle_message read tag failed 2024-03-29T11:39:58.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.400+0000 7f134ffff640 1 --1- 172.21.15.155:0/593510316 >> v1:172.21.15.155:6789/0 conn(0x7f1350145ba0 0x7f135014d830 :-1 s=OPENED pgs=528383 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:39:58.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.400+0000 7f134e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.155:6789/0 2024-03-29T11:39:58.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.404+0000 7f134e7fc640 1 -- 172.21.15.155:0/593510316 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288240242 0 0) 0x7f13500aa260 con 0x7f1350071f40 2024-03-29T11:39:58.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.404+0000 7f134e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:58.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.404+0000 7f134e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:58.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.404+0000 7f134e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:58.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.404+0000 7f134e7fc640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:39:58.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.404+0000 7f134e7fc640 1 -- 172.21.15.155:0/593510316 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1338003520 con 0x7f1350071f40 2024-03-29T11:39:58.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.404+0000 7f134e7fc640 1 -- 172.21.15.155:0/593510316 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3726713631 0 0) 0x7f1338003520 con 0x7f1350071f40 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.404+0000 7f134e7fc640 1 -- 172.21.15.155:0/593510316 >> v1:172.21.15.155:6790/0 conn(0x7f135013f060 legacy=0x7f1350145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.404+0000 7f134e7fc640 1 -- 172.21.15.155:0/593510316 >> v1:172.21.15.155:6789/0 conn(0x7f1350145ba0 legacy=0x7f135014d830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 1 -- 172.21.15.155:0/593510316 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13500a9d40 con 0x7f1350071f40 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 1 -- 172.21.15.155:0/593510316 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1348001a10 con 0x7f1350071f40 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 10 monclient: dump: 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.405+0000 7f134e7fc640 1 -- 172.21.15.155:0/593510316 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1348001b70 con 0x7f1350071f40 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.406+0000 7f134e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:58.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.406+0000 7f134e7fc640 1 -- 172.21.15.155:0/593510316 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1348001fc0 con 0x7f1350071f40 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.406+0000 7f134e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.406+0000 7f134e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.406+0000 7f134e7fc640 10 monclient: dump: 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.406+0000 7f1355d56640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:39:58.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.406+0000 7f1355d56640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:58.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.406+0000 7f1355d56640 4 set_mon_vals no callback set 2024-03-29T11:39:58.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.407+0000 7f1355d56640 10 monclient: shutdown 2024-03-29T11:39:58.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.407+0000 7f1355d56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:58.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.407+0000 7f1355d56640 1 -- 172.21.15.155:0/593510316 >> v1:172.21.15.183:6789/0 conn(0x7f1350071f40 legacy=0x7f13500a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:58.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.411+0000 7f1355d56640 1 -- 172.21.15.155:0/593510316 shutdown_connections 2024-03-29T11:39:58.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.411+0000 7f1355d56640 1 -- 172.21.15.155:0/593510316 >> 172.21.15.155:0/593510316 conn(0x7f13500b3a00 msgr2=0x7f13500b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:58.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.412+0000 7f1355d56640 1 -- 172.21.15.155:0/593510316 shutdown_connections 2024-03-29T11:39:58.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.413+0000 7f1355d56640 1 -- 172.21.15.155:0/593510316 wait complete. 2024-03-29T11:39:58.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.413+0000 7f1355d56640 10 monclient: build_initial_monmap 2024-03-29T11:39:58.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.413+0000 7f1355d56640 10 monclient: monmap: 2024-03-29T11:39:58.414 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:39:58.415 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:58.415 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:39:58.415 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:39:58.415 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:58.415 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:58.415 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:39:58.415 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:39:58.415 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:39:58.415 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:58.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.414+0000 7f1355d56640 1 librados: starting msgr at 2024-03-29T11:39:58.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.414+0000 7f1355d56640 1 librados: starting objecter 2024-03-29T11:39:58.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.414+0000 7f1355d56640 1 Processor -- start 2024-03-29T11:39:58.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.414+0000 7f1355d56640 1 -- start start 2024-03-29T11:39:58.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.414+0000 7f1355d56640 1 librados: setting wanted keys 2024-03-29T11:39:58.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.415+0000 7f1355d56640 1 librados: calling monclient init 2024-03-29T11:39:58.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.415+0000 7f1355d56640 10 monclient: init 2024-03-29T11:39:58.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.415+0000 7f1355d56640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:58.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.416+0000 7f1355d56640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:58.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.416+0000 7f1355d56640 10 monclient(hunting): picked mon.noname-b con 0x7f1350071f40 addr v1:172.21.15.183:6789/0 2024-03-29T11:39:58.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.416+0000 7f1355d56640 10 monclient(hunting): picked mon.noname-c con 0x7f135013f060 addr v1:172.21.15.155:6790/0 2024-03-29T11:39:58.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.416+0000 7f1355d56640 10 monclient(hunting): picked mon.noname-a con 0x7f1350145ba0 addr v1:172.21.15.155:6789/0 2024-03-29T11:39:58.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.416+0000 7f1355d56640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1350144b40 con 0x7f1350145ba0 2024-03-29T11:39:58.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.416+0000 7f1355d56640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1350140a30 con 0x7f1350071f40 2024-03-29T11:39:58.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.416+0000 7f1355d56640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13500ac260 con 0x7f135013f060 2024-03-29T11:39:58.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.416+0000 7f1355d56640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:58.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.416+0000 7f1355d56640 10 monclient(hunting): authenticate will time out at 1180.542375s 2024-03-29T11:39:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.420+0000 7f134f7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f1350071f40 0x7f1350145340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:35297/0 (socket says 172.21.15.155:35297) 2024-03-29T11:39:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.420+0000 7f134ffff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f1350145ba0 0x7f13501e1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:47670/0 (socket says 172.21.15.155:47670) 2024-03-29T11:39:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.420+0000 7f134ffff640 1 -- 172.21.15.155:0/2650725303 learned_addr learned my addr 172.21.15.155:0/2650725303 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:39:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.421+0000 7f1337fff640 10 client.?.objecter ms_handle_connect 0x7f1350145ba0 2024-03-29T11:39:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.421+0000 7f1337fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.421+0000 7f1337fff640 10 client.?.objecter ms_handle_connect 0x7f1350071f40 2024-03-29T11:39:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.421+0000 7f1337fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.421+0000 7f1337fff640 10 client.?.objecter ms_handle_connect 0x7f135013f060 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.421+0000 7f1337fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.421+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128372003 0 0) 0x7f1350140a30 con 0x7f1350071f40 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.421+0000 7f1337fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.421+0000 7f1337fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.421+0000 7f1337fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133c004eb0 con 0x7f1350071f40 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485914460 0 0) 0x7f1350144b40 con 0x7f1350145ba0 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:39:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133c006230 con 0x7f1350145ba0 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705909626 0 0) 0x7f13500ac260 con 0x7f135013f060 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1350144b40 con 0x7f135013f060 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.422+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 13725594 0 0) 0x7f133c006230 con 0x7f1350145ba0 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.423+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f133c006830 con 0x7f1350145ba0 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.423+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1344001810 con 0x7f1350145ba0 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.423+0000 7f1337fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.423+0000 7f1337fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.423+0000 7f1337fff640 10 monclient(hunting): dump: 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:58.427 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.423+0000 7f1337fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.423+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2825378562 0 0) 0x7f133c004eb0 con 0x7f1350071f40 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.423+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f133c006230 con 0x7f1350071f40 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.423+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1348001f40 con 0x7f1350071f40 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.423+0000 7f1337fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.423+0000 7f1337fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:58.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.424+0000 7f1337fff640 10 monclient(hunting): dump: 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.424+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2836569657 0 0) 0x7f133c006830 con 0x7f1350145ba0 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.424+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 >> v1:172.21.15.155:6790/0 conn(0x7f135013f060 legacy=0x7f1350144430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.424+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 >> v1:172.21.15.183:6789/0 conn(0x7f1350071f40 legacy=0x7f1350145340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.424+0000 7f1337fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:58.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.424+0000 7f1337fff640 1 monclient: found mon.a 2024-03-29T11:39:58.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.425+0000 7f1337fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:58.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.425+0000 7f1337fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:39:58.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.425+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13500ac3f0 con 0x7f1350145ba0 2024-03-29T11:39:58.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.425+0000 7f1337fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:58.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.425+0000 7f1337fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.425+0000 7f1337fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.426+0000 7f1355d56640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.426+0000 7f1355d56640 10 monclient: _renew_subs 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.426+0000 7f1355d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.426+0000 7f1355d56640 1 -- 172.21.15.155:0/2650725303 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13501d7400 con 0x7f1350145ba0 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.426+0000 7f1355d56640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.426+0000 7f1355d56640 10 monclient: _renew_subs 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.427+0000 7f1355d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.427+0000 7f1355d56640 1 -- 172.21.15.155:0/2650725303 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13501e3560 con 0x7f1350145ba0 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.427+0000 7f1355d56640 1 librados: init done 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.428+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1344001c40 con 0x7f1350145ba0 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.428+0000 7f1337fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:58.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.428+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1344004980 con 0x7f1350145ba0 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.428+0000 7f1337fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.428+0000 7f1337fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.428+0000 7f1337fff640 10 monclient: dump: 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:39:58.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.428+0000 7f134dffb640 4 set_mon_vals no callback set 2024-03-29T11:39:58.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.428+0000 7f1355d56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:58.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.429+0000 7f1355d56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:58.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.429+0000 7f1355d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:39:58.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.429+0000 7f1355d56640 1 -- 172.21.15.155:0/2650725303 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1350145060 con 0x7f1350145ba0 2024-03-29T11:39:58.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.431+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f1344004ae0 con 0x7f1350145ba0 2024-03-29T11:39:58.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.432+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f13440849d0 con 0x7f1350145ba0 2024-03-29T11:39:58.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.432+0000 7f1337fff640 10 client.4419.objecter ms_dispatch 0x7f1350000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:58.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.432+0000 7f1337fff640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:58.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.432+0000 7f1337fff640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:58.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.432+0000 7f1337fff640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-29T11:39:58.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.433+0000 7f134f7fe640 10 monclient: get_auth_request con 0x7f133c06e490 auth_method 0 2024-03-29T11:39:58.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.434+0000 7f1337fff640 10 client.4419.objecter ms_handle_connect 0x7f133c06e490 2024-03-29T11:39:58.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.440+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1350145060 con 0x7f1350145ba0 2024-03-29T11:39:58.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.440+0000 7f1337fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:58.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.440+0000 7f1337fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:58.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.777+0000 7f1355d56640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:58.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.777+0000 7f1355d56640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:58.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.777+0000 7f1355d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:39:58.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.777+0000 7f1355d56640 1 -- 172.21.15.155:0/2650725303 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1350145060 con 0x7f1350145ba0 2024-03-29T11:39:58.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.779+0000 7f1337fff640 1 -- 172.21.15.155:0/2650725303 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f1350145060 con 0x7f1350145ba0 2024-03-29T11:39:58.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.779+0000 7f1337fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:58.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.779+0000 7f1337fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:58.781 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:39:58.781 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:58.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.783+0000 7f1355d56640 10 librados: watch_flush enter 2024-03-29T11:39:58.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.783+0000 7f1355d56640 10 librados: watch_flush exit 2024-03-29T11:39:58.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.783+0000 7f1355d56640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-29T11:39:58.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.783+0000 7f1355d56640 10 client.4419.objecter successfully canceled tick 2024-03-29T11:39:58.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.783+0000 7f1355d56640 1 -- 172.21.15.155:0/2650725303 >> v1:172.21.15.155:6816/1760814708 conn(0x7f133c06e490 legacy=0x7f133c070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:58.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.783+0000 7f1355d56640 10 monclient: shutdown 2024-03-29T11:39:58.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.783+0000 7f1355d56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:58.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.783+0000 7f1355d56640 1 -- 172.21.15.155:0/2650725303 >> v1:172.21.15.155:6789/0 conn(0x7f1350145ba0 legacy=0x7f13501e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:58.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.783+0000 7f1355d56640 1 -- 172.21.15.155:0/2650725303 shutdown_connections 2024-03-29T11:39:58.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.783+0000 7f1355d56640 1 -- 172.21.15.155:0/2650725303 >> 172.21.15.155:0/2650725303 conn(0x7f13500b3a00 msgr2=0x7f13501396e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:58.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.783+0000 7f1355d56640 1 -- 172.21.15.155:0/2650725303 shutdown_connections 2024-03-29T11:39:58.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.784+0000 7f1355d56640 1 -- 172.21.15.155:0/2650725303 wait complete. 2024-03-29T11:39:58.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:39:58.784+0000 7f1355d56640 1 librados: shutdown 2024-03-29T11:39:58.802 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:05.608 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:05.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.760+0000 7f313c4ba640 10 monclient: get_monmap_and_config 2024-03-29T11:40:05.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.760+0000 7f313c4ba640 10 monclient: build_initial_monmap 2024-03-29T11:40:05.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.760+0000 7f313c4ba640 10 monclient: monmap: 2024-03-29T11:40:05.763 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:05.763 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:05.763 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:40:05.761763+0000 2024-03-29T11:40:05.763 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:40:05.761763+0000 2024-03-29T11:40:05.763 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:05.763 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:05.763 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:05.763 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:40:05.763 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:40:05.763 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:05.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.761+0000 7f313c4ba640 1 Processor -- start 2024-03-29T11:40:05.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.761+0000 7f313c4ba640 1 -- start start 2024-03-29T11:40:05.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.761+0000 7f313c4ba640 10 monclient: init 2024-03-29T11:40:05.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f313c4ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:05.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f313c4ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:05.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f313c4ba640 10 monclient(hunting): picked mon.noname-a con 0x7f313406d9d0 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:05.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f313c4ba640 10 monclient(hunting): picked mon.noname-b con 0x7f313413e820 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:05.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f313c4ba640 10 monclient(hunting): picked mon.noname-c con 0x7f31340af1d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:05.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f313c4ba640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31340aed10 con 0x7f313406d9d0 2024-03-29T11:40:05.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f313c4ba640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31340afd50 con 0x7f31340af1d0 2024-03-29T11:40:05.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f313c4ba640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31340ab490 con 0x7f313413e820 2024-03-29T11:40:05.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f313c4ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:05.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f313c4ba640 10 monclient(hunting): authenticate will time out at 1187.889537s 2024-03-29T11:40:05.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f3139a55640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f313413e820 0x7f313413ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39466/0 (socket says 172.21.15.155:39466) 2024-03-29T11:40:05.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.762+0000 7f3139a55640 1 -- 172.21.15.155:0/1023775898 learned_addr learned my addr 172.21.15.155:0/1023775898 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 1 -- 172.21.15.155:0/1023775898 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228828382 0 0) 0x7f31340ab490 con 0x7f313413e820 2024-03-29T11:40:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 10 monclient(hunting): _negotiate 2024-03-29T11:40:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:40:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 1 -- 172.21.15.155:0/1023775898 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3120003520 con 0x7f313413e820 2024-03-29T11:40:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 1 -- 172.21.15.155:0/1023775898 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146001790 0 0) 0x7f31340aed10 con 0x7f313406d9d0 2024-03-29T11:40:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 10 monclient(hunting): _negotiate 2024-03-29T11:40:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:05.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:40:05.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 1 -- 172.21.15.155:0/1023775898 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31340ab490 con 0x7f313406d9d0 2024-03-29T11:40:05.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 1 -- 172.21.15.155:0/1023775898 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 374870549 0 0) 0x7f31340ab490 con 0x7f313406d9d0 2024-03-29T11:40:05.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 1 -- 172.21.15.155:0/1023775898 >> v1:172.21.15.155:6790/0 conn(0x7f313413e820 legacy=0x7f313413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:05.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.763+0000 7f3139254640 1 -- 172.21.15.155:0/1023775898 >> v1:172.21.15.183:6789/0 conn(0x7f31340af1d0 legacy=0x7f31340af640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 10 monclient: _finish_hunting 0 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 1 monclient: found mon.noname-a 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 1 -- 172.21.15.155:0/1023775898 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31340ab770 con 0x7f313406d9d0 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 10 monclient: _finish_auth 0 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 10 monclient: _check_auth_tickets 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 1 -- 172.21.15.155:0/1023775898 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f312c001920 con 0x7f313406d9d0 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f3139254640 10 monclient: dump: 2024-03-29T11:40:05.768 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:05.769 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:05.769 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:05.769 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:05.769 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:05.769 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:05.769 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:05.769 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:05.769 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:05.769 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:05.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f313c4ba640 5 monclient: authenticate success, global_id 4422 2024-03-29T11:40:05.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.765+0000 7f313c4ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:05.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.766+0000 7f3139254640 1 -- 172.21.15.155:0/1023775898 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f312c001b00 con 0x7f313406d9d0 2024-03-29T11:40:05.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.766+0000 7f3139254640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:05.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.766+0000 7f3139254640 1 -- 172.21.15.155:0/1023775898 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f312c001f50 con 0x7f313406d9d0 2024-03-29T11:40:05.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.766+0000 7f313c4ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:05.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.766+0000 7f313c4ba640 4 set_mon_vals no callback set 2024-03-29T11:40:05.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.766+0000 7f3139254640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:05.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.766+0000 7f3139254640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:05.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.766+0000 7f3139254640 10 monclient: dump: 2024-03-29T11:40:05.776 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:05.776 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.767+0000 7f313c4ba640 10 monclient: shutdown 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.767+0000 7f313c4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.767+0000 7f313c4ba640 1 -- 172.21.15.155:0/1023775898 >> v1:172.21.15.155:6789/0 conn(0x7f313406d9d0 legacy=0x7f31340b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.767+0000 7f313c4ba640 1 -- 172.21.15.155:0/1023775898 shutdown_connections 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.767+0000 7f313c4ba640 1 -- 172.21.15.155:0/1023775898 >> 172.21.15.155:0/1023775898 conn(0x7f313406b660 msgr2=0x7f313406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:05.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.767+0000 7f313c4ba640 1 -- 172.21.15.155:0/1023775898 shutdown_connections 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.767+0000 7f313c4ba640 1 -- 172.21.15.155:0/1023775898 wait complete. 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.768+0000 7f313c4ba640 10 monclient: build_initial_monmap 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.768+0000 7f313c4ba640 10 monclient: monmap: 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:05.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.769+0000 7f313c4ba640 1 librados: starting msgr at 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.769+0000 7f313c4ba640 1 librados: starting objecter 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.769+0000 7f313c4ba640 1 Processor -- start 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.769+0000 7f313c4ba640 1 -- start start 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.769+0000 7f313c4ba640 1 librados: setting wanted keys 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.769+0000 7f313c4ba640 1 librados: calling monclient init 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.769+0000 7f313c4ba640 10 monclient: init 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f313c4ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f313c4ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f313c4ba640 10 monclient(hunting): picked mon.noname-b con 0x7f313406d9d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f313c4ba640 10 monclient(hunting): picked mon.noname-a con 0x7f31340af1d0 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f313c4ba640 10 monclient(hunting): picked mon.noname-c con 0x7f313413e820 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f313c4ba640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3134071110 con 0x7f31340af1d0 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f313c4ba640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313406f790 con 0x7f313406d9d0 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f3139a55640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f31340af1d0 0x7f3134136410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:33788/0 (socket says 172.21.15.155:33788) 2024-03-29T11:40:05.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f3139a55640 1 -- 172.21.15.155:0/1616172214 learned_addr learned my addr 172.21.15.155:0/1616172214 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f313c4ba640 1 -- 172.21.15.155:0/1616172214 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31340713d0 con 0x7f313413e820 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f313c4ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f313c4ba640 10 monclient(hunting): authenticate will time out at 1187.897461s 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f312affd640 10 client.?.objecter ms_handle_connect 0x7f31340af1d0 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f312affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f312affd640 10 client.?.objecter ms_handle_connect 0x7f313413e820 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f312affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203947454 0 0) 0x7f3134071110 con 0x7f31340af1d0 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f312affd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f312affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f312affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f312affd640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.770+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3118004eb0 con 0x7f31340af1d0 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424787211 0 0) 0x7f31340713d0 con 0x7f313413e820 2024-03-29T11:40:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3134071110 con 0x7f313413e820 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2253561934 0 0) 0x7f3134071110 con 0x7f313413e820 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31340713d0 con 0x7f313413e820 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f31240017b0 con 0x7f313413e820 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 10 monclient(hunting): dump: 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:05.781 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2693960830 0 0) 0x7f3118004eb0 con 0x7f31340af1d0 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3134071110 con 0x7f31340af1d0 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3130001950 con 0x7f31340af1d0 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 10 monclient(hunting): dump: 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:05.782 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2622155974 0 0) 0x7f31340713d0 con 0x7f313413e820 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.771+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 >> v1:172.21.15.183:6789/0 conn(0x7f313406d9d0 legacy=0x7f3134135d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 >> v1:172.21.15.155:6789/0 conn(0x7f31340af1d0 legacy=0x7f3134136410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 1 monclient: found mon.c 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:05.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313406fa00 con 0x7f313413e820 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 10 monclient: _finish_auth 0 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f313c4ba640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f313c4ba640 10 monclient: _renew_subs 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f313c4ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f313c4ba640 1 -- 172.21.15.155:0/1616172214 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313406fce0 con 0x7f313413e820 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f313c4ba640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f313c4ba640 10 monclient: _renew_subs 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f313c4ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f313c4ba640 1 -- 172.21.15.155:0/1616172214 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f313406de60 con 0x7f313413e820 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3124001910 con 0x7f313413e820 2024-03-29T11:40:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f313c4ba640 1 librados: init done 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3124003830 con 0x7f313413e820 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f312affd640 10 monclient: dump: 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:05.785 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:05.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.772+0000 7f3138a53640 4 set_mon_vals no callback set 2024-03-29T11:40:05.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.774+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f3124019800 con 0x7f313413e820 2024-03-29T11:40:05.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.775+0000 7f313c4ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:05.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.775+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f3124083750 con 0x7f313413e820 2024-03-29T11:40:05.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.775+0000 7f312affd640 10 client.4451.objecter ms_dispatch 0x7f3134000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:05.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.775+0000 7f312affd640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:05.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.775+0000 7f312affd640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:05.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.775+0000 7f313c4ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:05.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.775+0000 7f313c4ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:05.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.775+0000 7f313c4ba640 1 -- 172.21.15.155:0/1616172214 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313413e2f0 con 0x7f313413e820 2024-03-29T11:40:05.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.775+0000 7f312affd640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:40:05.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.775+0000 7f313a256640 10 monclient: get_auth_request con 0x7f311806de40 auth_method 0 2024-03-29T11:40:05.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.776+0000 7f312affd640 10 client.4451.objecter ms_handle_connect 0x7f311806de40 2024-03-29T11:40:05.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.783+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f312404f030 con 0x7f313413e820 2024-03-29T11:40:05.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.783+0000 7f312affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:05.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:05.783+0000 7f312affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:06.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.123+0000 7f313c4ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:06.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.123+0000 7f313c4ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:06.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.123+0000 7f313c4ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:06.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.123+0000 7f313c4ba640 1 -- 172.21.15.155:0/1616172214 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3134075f10 con 0x7f313413e820 2024-03-29T11:40:06.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.124+0000 7f312affd640 1 -- 172.21.15.155:0/1616172214 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f3124050f60 con 0x7f313413e820 2024-03-29T11:40:06.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.124+0000 7f312affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:06.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.124+0000 7f312affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:06.127 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:40:06.127 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:06.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.128+0000 7f313c4ba640 10 librados: watch_flush enter 2024-03-29T11:40:06.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.128+0000 7f313c4ba640 10 librados: watch_flush exit 2024-03-29T11:40:06.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.128+0000 7f313c4ba640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:40:06.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.128+0000 7f313c4ba640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:40:06.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.129+0000 7f313c4ba640 1 -- 172.21.15.155:0/1616172214 >> v1:172.21.15.155:6816/1760814708 conn(0x7f311806de40 legacy=0x7f3118070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:06.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.129+0000 7f313c4ba640 10 monclient: shutdown 2024-03-29T11:40:06.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.129+0000 7f313c4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:06.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.129+0000 7f313c4ba640 1 -- 172.21.15.155:0/1616172214 >> v1:172.21.15.155:6790/0 conn(0x7f313413e820 legacy=0x7f31341e6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:06.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.129+0000 7f313c4ba640 1 -- 172.21.15.155:0/1616172214 shutdown_connections 2024-03-29T11:40:06.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.129+0000 7f313c4ba640 1 -- 172.21.15.155:0/1616172214 >> 172.21.15.155:0/1616172214 conn(0x7f313406b660 msgr2=0x7f313414d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:06.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.129+0000 7f313c4ba640 1 -- 172.21.15.155:0/1616172214 shutdown_connections 2024-03-29T11:40:06.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.130+0000 7f313c4ba640 1 -- 172.21.15.155:0/1616172214 wait complete. 2024-03-29T11:40:06.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:06.130+0000 7f313c4ba640 1 librados: shutdown 2024-03-29T11:40:06.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:12.954 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:13.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.086+0000 7fb7c260f640 10 monclient: get_monmap_and_config 2024-03-29T11:40:13.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.086+0000 7fb7c260f640 10 monclient: build_initial_monmap 2024-03-29T11:40:13.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.086+0000 7fb7c260f640 10 monclient: monmap: 2024-03-29T11:40:13.088 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:13.088 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:13.088 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:40:13.087593+0000 2024-03-29T11:40:13.088 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:40:13.087593+0000 2024-03-29T11:40:13.088 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:13.088 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:13.089 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:13.089 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:40:13.089 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:40:13.089 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.086+0000 7fb7c260f640 1 Processor -- start 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.087+0000 7fb7c260f640 1 -- start start 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.087+0000 7fb7c260f640 10 monclient: init 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.087+0000 7fb7c260f640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.087+0000 7fb7c260f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.087+0000 7fb7c260f640 10 monclient(hunting): picked mon.noname-b con 0x7fb7bc0a9750 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.087+0000 7fb7bbfff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fb7bc0a9750 0x7fb7bc13dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:34952/0 (socket says 172.21.15.155:34952) 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7c260f640 10 monclient(hunting): picked mon.noname-a con 0x7fb7bc13e1e0 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7bbfff640 1 -- 172.21.15.155:0/3764456332 learned_addr learned my addr 172.21.15.155:0/3764456332 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7c260f640 10 monclient(hunting): picked mon.noname-c con 0x7fb7bc071c30 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7c260f640 1 -- 172.21.15.155:0/3764456332 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc0ab300 con 0x7fb7bc13e1e0 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7c260f640 1 -- 172.21.15.155:0/3764456332 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc0ac2f0 con 0x7fb7bc071c30 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7c260f640 1 -- 172.21.15.155:0/3764456332 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc0ad8d0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7c260f640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7c260f640 10 monclient(hunting): authenticate will time out at 1195.215251s 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 1 -- 172.21.15.155:0/3764456332 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466456420 0 0) 0x7fb7bc0ac2f0 con 0x7fb7bc071c30 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 1 -- 172.21.15.155:0/3764456332 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a4003520 con 0x7fb7bc071c30 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 1 -- 172.21.15.155:0/3764456332 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616255778 0 0) 0x7fb7bc0ad8d0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 1 -- 172.21.15.155:0/3764456332 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7bc0ac2f0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 1 -- 172.21.15.155:0/3764456332 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3116863372 0 0) 0x7fb7bc0ac2f0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 1 -- 172.21.15.155:0/3764456332 >> v1:172.21.15.183:6789/0 conn(0x7fb7bc071c30 legacy=0x7fb7bc0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.088+0000 7fb7baffd640 1 -- 172.21.15.155:0/3764456332 >> v1:172.21.15.155:6789/0 conn(0x7fb7bc13e1e0 legacy=0x7fb7bc0ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:13.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:13.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 1 monclient: found mon.noname-b 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 1 -- 172.21.15.155:0/3764456332 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc0ada60 con 0x7fb7bc0a9750 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 1 -- 172.21.15.155:0/3764456332 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb7ac001670 con 0x7fb7bc0a9750 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 10 monclient: dump: 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:13.095 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7c260f640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7c260f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 1 -- 172.21.15.155:0/3764456332 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7ac0017d0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 1 -- 172.21.15.155:0/3764456332 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb7ac001be0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.092+0000 7fb7baffd640 10 monclient: dump: 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:13.096 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:13.097 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:13.097 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:13.097 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:13.097 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:13.097 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:13.097 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:13.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.093+0000 7fb7c260f640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.093+0000 7fb7c260f640 4 set_mon_vals no callback set 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.093+0000 7fb7c260f640 10 monclient: shutdown 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.094+0000 7fb7c260f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.094+0000 7fb7c260f640 1 -- 172.21.15.155:0/3764456332 >> v1:172.21.15.155:6790/0 conn(0x7fb7bc0a9750 legacy=0x7fb7bc13dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.094+0000 7fb7c260f640 1 -- 172.21.15.155:0/3764456332 shutdown_connections 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.094+0000 7fb7c260f640 1 -- 172.21.15.155:0/3764456332 >> 172.21.15.155:0/3764456332 conn(0x7fb7bc06b660 msgr2=0x7fb7bc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.094+0000 7fb7c260f640 1 -- 172.21.15.155:0/3764456332 shutdown_connections 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.094+0000 7fb7c260f640 1 -- 172.21.15.155:0/3764456332 wait complete. 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.095+0000 7fb7c260f640 10 monclient: build_initial_monmap 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.095+0000 7fb7c260f640 10 monclient: monmap: 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:13.099 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.095+0000 7fb7c260f640 1 librados: starting msgr at 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.095+0000 7fb7c260f640 1 librados: starting objecter 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.095+0000 7fb7c260f640 1 Processor -- start 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.096+0000 7fb7c260f640 1 -- start start 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.096+0000 7fb7c260f640 1 librados: setting wanted keys 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.096+0000 7fb7c260f640 1 librados: calling monclient init 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.096+0000 7fb7c260f640 10 monclient: init 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.097+0000 7fb7c260f640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:13.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.097+0000 7fb7c260f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:13.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.097+0000 7fb7c260f640 10 monclient(hunting): picked mon.noname-a con 0x7fb7bc071c30 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:13.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.097+0000 7fb7c260f640 10 monclient(hunting): picked mon.noname-c con 0x7fb7bc0a9750 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:13.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7c260f640 10 monclient(hunting): picked mon.noname-b con 0x7fb7bc13e1e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:13.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7c260f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc070cd0 con 0x7fb7bc071c30 2024-03-29T11:40:13.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7c260f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc06e6f0 con 0x7fb7bc13e1e0 2024-03-29T11:40:13.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7c260f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc06e880 con 0x7fb7bc0a9750 2024-03-29T11:40:13.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7c260f640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:13.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7c260f640 10 monclient(hunting): authenticate will time out at 1195.225096s 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7bb7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fb7bc0a9750 0x7fb7bc06feb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:37710/0 (socket says 172.21.15.155:37710) 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7bb7fe640 1 -- 172.21.15.155:0/2354178561 learned_addr learned my addr 172.21.15.155:0/2354178561 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 10 client.?.objecter ms_handle_connect 0x7fb7bc0a9750 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 10 client.?.objecter ms_handle_connect 0x7fb7bc13e1e0 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964436142 0 0) 0x7fb7bc06e880 con 0x7fb7bc0a9750 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a8004eb0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443624892 0 0) 0x7fb7bc06e6f0 con 0x7fb7bc13e1e0 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:13.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.098+0000 7fb7b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7bc06e880 con 0x7fb7bc13e1e0 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1610153243 0 0) 0x7fb7a8004eb0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7bc06e6f0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb7b00018f0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:13.108 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3900460917 0 0) 0x7fb7bc06e6f0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 >> v1:172.21.15.183:6789/0 conn(0x7fb7bc13e1e0 legacy=0x7fb7bc0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 >> v1:172.21.15.155:6789/0 conn(0x7fb7bc071c30 legacy=0x7fb7bc06dfe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.099+0000 7fb7b8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 1 monclient: found mon.c 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc1d6520 con 0x7fb7bc0a9750 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:13.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7c260f640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7c260f640 10 monclient: _renew_subs 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7b0001a50 con 0x7fb7bc0a9750 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7c260f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7c260f640 1 -- 172.21.15.155:0/2354178561 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7bc071560 con 0x7fb7bc0a9750 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb7b0003870 con 0x7fb7bc0a9750 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7b8ff9640 10 monclient: dump: 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:13.110 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.100+0000 7fb7ba7fc640 4 set_mon_vals no callback set 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.101+0000 7fb7c260f640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.101+0000 7fb7c260f640 10 monclient: _renew_subs 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.101+0000 7fb7c260f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.101+0000 7fb7c260f640 1 -- 172.21.15.155:0/2354178561 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7bc13c0a0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.101+0000 7fb7c260f640 1 librados: init done 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.102+0000 7fb7a27fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.102+0000 7fb7a27fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.102+0000 7fb7a27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.102+0000 7fb7a27fc640 1 -- 172.21.15.155:0/2354178561 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7bc0abb00 con 0x7fb7bc0a9750 2024-03-29T11:40:13.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.103+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fb7b00039d0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.103+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fb7b00849c0 con 0x7fb7bc0a9750 2024-03-29T11:40:13.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.103+0000 7fb7b8ff9640 10 client.4457.objecter ms_dispatch 0x7fb7bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:13.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.103+0000 7fb7b8ff9640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:13.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.103+0000 7fb7b8ff9640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:13.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.103+0000 7fb7b8ff9640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-29T11:40:13.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.105+0000 7fb7bbfff640 10 monclient: get_auth_request con 0x7fb7a806dcb0 auth_method 0 2024-03-29T11:40:13.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.105+0000 7fb7b8ff9640 10 client.4457.objecter ms_handle_connect 0x7fb7a806dcb0 2024-03-29T11:40:13.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.110+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7b0050300 con 0x7fb7bc0a9750 2024-03-29T11:40:13.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.110+0000 7fb7b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:13.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.110+0000 7fb7b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:13.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.422+0000 7fb7a27fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:13.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.422+0000 7fb7a27fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:13.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.422+0000 7fb7a27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:13.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.422+0000 7fb7a27fc640 1 -- 172.21.15.155:0/2354178561 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7bc0a9b30 con 0x7fb7bc0a9750 2024-03-29T11:40:13.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.423+0000 7fb7b8ff9640 1 -- 172.21.15.155:0/2354178561 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fb7b0052230 con 0x7fb7bc0a9750 2024-03-29T11:40:13.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.423+0000 7fb7b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:13.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.423+0000 7fb7b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:13.426 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:40:13.427 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:13.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 10 librados: watch_flush enter 2024-03-29T11:40:13.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 10 librados: watch_flush exit 2024-03-29T11:40:13.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-29T11:40:13.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 10 client.4457.objecter successfully canceled tick 2024-03-29T11:40:13.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 1 -- 172.21.15.155:0/2354178561 >> v1:172.21.15.155:6816/1760814708 conn(0x7fb7a806dcb0 legacy=0x7fb7a8070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 10 monclient: shutdown 2024-03-29T11:40:13.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:13.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 1 -- 172.21.15.155:0/2354178561 >> v1:172.21.15.155:6790/0 conn(0x7fb7bc0a9750 legacy=0x7fb7bc06feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 1 -- 172.21.15.155:0/2354178561 shutdown_connections 2024-03-29T11:40:13.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 1 -- 172.21.15.155:0/2354178561 >> 172.21.15.155:0/2354178561 conn(0x7fb7bc06b660 msgr2=0x7fb7bc0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:13.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 1 -- 172.21.15.155:0/2354178561 shutdown_connections 2024-03-29T11:40:13.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 1 -- 172.21.15.155:0/2354178561 wait complete. 2024-03-29T11:40:13.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:13.426+0000 7fb7a27fc640 1 librados: shutdown 2024-03-29T11:40:13.443 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:20.250 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.399+0000 7fa7ba06d640 10 monclient: get_monmap_and_config 2024-03-29T11:40:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.399+0000 7fa7ba06d640 10 monclient: build_initial_monmap 2024-03-29T11:40:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.399+0000 7fa7ba06d640 10 monclient: monmap: 2024-03-29T11:40:20.402 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:20.402 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:20.402 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:40:20.401251+0000 2024-03-29T11:40:20.402 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:40:20.401251+0000 2024-03-29T11:40:20.402 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:20.402 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:20.402 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:20.403 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:40:20.403 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:40:20.403 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:20.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.400+0000 7fa7ba06d640 1 Processor -- start 2024-03-29T11:40:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 1 -- start start 2024-03-29T11:40:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 10 monclient: init 2024-03-29T11:40:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 10 monclient(hunting): picked mon.noname-c con 0x7fa7ac0389e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 10 monclient(hunting): picked mon.noname-b con 0x7fa7ac038dc0 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 10 monclient(hunting): picked mon.noname-a con 0x7fa7ac030660 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac0c54c0 con 0x7fa7ac030660 2024-03-29T11:40:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac036430 con 0x7fa7ac0389e0 2024-03-29T11:40:20.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac039840 con 0x7fa7ac038dc0 2024-03-29T11:40:20.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:20.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.401+0000 7fa7ba06d640 10 monclient(hunting): authenticate will time out at 1202.528812s 2024-03-29T11:40:20.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.403+0000 7fa7b886a640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa7ac038dc0 0x7fa7ac0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:47604/0 (socket says 172.21.15.155:47604) 2024-03-29T11:40:20.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.403+0000 7fa7b886a640 1 -- 172.21.15.155:0/2161490674 learned_addr learned my addr 172.21.15.155:0/2161490674 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:20.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.404+0000 7fa7a7fff640 1 -- 172.21.15.155:0/2161490674 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927371795 0 0) 0x7fa7ac039840 con 0x7fa7ac038dc0 2024-03-29T11:40:20.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.404+0000 7fa7a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:20.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.404+0000 7fa7a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:20.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.404+0000 7fa7a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:20.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.404+0000 7fa7a7fff640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:40:20.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.404+0000 7fa7a7fff640 1 -- 172.21.15.155:0/2161490674 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a0003520 con 0x7fa7ac038dc0 2024-03-29T11:40:20.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 1 -- 172.21.15.155:0/2161490674 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3598600946 0 0) 0x7fa7a0003520 con 0x7fa7ac038dc0 2024-03-29T11:40:20.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 1 -- 172.21.15.155:0/2161490674 >> v1:172.21.15.183:6789/0 conn(0x7fa7ac0389e0 legacy=0x7fa7ac033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:20.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 1 -- 172.21.15.155:0/2161490674 >> v1:172.21.15.155:6789/0 conn(0x7fa7ac030660 legacy=0x7fa7ac0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:20.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:20.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 1 monclient: found mon.noname-b 2024-03-29T11:40:20.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:20.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 1 -- 172.21.15.155:0/2161490674 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac035690 con 0x7fa7ac038dc0 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 10 monclient: _finish_auth 0 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 1 -- 172.21.15.155:0/2161490674 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa7a8001690 con 0x7fa7ac038dc0 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 10 monclient: dump: 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:20.409 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:20.410 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:20.410 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:20.410 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:20.410 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:20.410 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 1 -- 172.21.15.155:0/2161490674 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7a8001860 con 0x7fa7ac038dc0 2024-03-29T11:40:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 1 -- 172.21.15.155:0/2161490674 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa7a8001cb0 con 0x7fa7ac038dc0 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.406+0000 7fa7a7fff640 10 monclient: dump: 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.408+0000 7fa7ba06d640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.408+0000 7fa7ba06d640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:20.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.408+0000 7fa7ba06d640 4 set_mon_vals no callback set 2024-03-29T11:40:20.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.409+0000 7fa7ba06d640 10 monclient: shutdown 2024-03-29T11:40:20.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.409+0000 7fa7ba06d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:20.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.409+0000 7fa7ba06d640 1 -- 172.21.15.155:0/2161490674 >> v1:172.21.15.155:6790/0 conn(0x7fa7ac038dc0 legacy=0x7fa7ac0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:20.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.409+0000 7fa7ba06d640 1 -- 172.21.15.155:0/2161490674 shutdown_connections 2024-03-29T11:40:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.409+0000 7fa7ba06d640 1 -- 172.21.15.155:0/2161490674 >> 172.21.15.155:0/2161490674 conn(0x7fa7ac005c00 msgr2=0x7fa7ac006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.409+0000 7fa7ba06d640 1 -- 172.21.15.155:0/2161490674 shutdown_connections 2024-03-29T11:40:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.409+0000 7fa7ba06d640 1 -- 172.21.15.155:0/2161490674 wait complete. 2024-03-29T11:40:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.410+0000 7fa7ba06d640 10 monclient: build_initial_monmap 2024-03-29T11:40:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.410+0000 7fa7ba06d640 10 monclient: monmap: 2024-03-29T11:40:20.413 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.411+0000 7fa7ba06d640 1 librados: starting msgr at 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.411+0000 7fa7ba06d640 1 librados: starting objecter 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.411+0000 7fa7ba06d640 1 Processor -- start 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.411+0000 7fa7ba06d640 1 -- start start 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.411+0000 7fa7ba06d640 1 librados: setting wanted keys 2024-03-29T11:40:20.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.411+0000 7fa7ba06d640 1 librados: calling monclient init 2024-03-29T11:40:20.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.411+0000 7fa7ba06d640 10 monclient: init 2024-03-29T11:40:20.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.412+0000 7fa7ba06d640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:20.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.412+0000 7fa7ba06d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:20.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.412+0000 7fa7ba06d640 10 monclient(hunting): picked mon.noname-a con 0x7fa7ac030660 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:20.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.412+0000 7fa7ba06d640 10 monclient(hunting): picked mon.noname-b con 0x7fa7ac0389e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:20.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.412+0000 7fa7ba06d640 10 monclient(hunting): picked mon.noname-c con 0x7fa7ac02fb70 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:20.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.412+0000 7fa7ba06d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac0c6b70 con 0x7fa7ac030660 2024-03-29T11:40:20.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.412+0000 7fa7ba06d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac0c6de0 con 0x7fa7ac0389e0 2024-03-29T11:40:20.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.412+0000 7fa7ba06d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac0c5ec0 con 0x7fa7ac02fb70 2024-03-29T11:40:20.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.412+0000 7fa7ba06d640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:20.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.412+0000 7fa7ba06d640 10 monclient(hunting): authenticate will time out at 1202.539600s 2024-03-29T11:40:20.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.415+0000 7fa7b986c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa7ac02fb70 0x7fa7ac167190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:49082/0 (socket says 172.21.15.155:49082) 2024-03-29T11:40:20.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.415+0000 7fa7b986c640 1 -- 172.21.15.155:0/1274051183 learned_addr learned my addr 172.21.15.155:0/1274051183 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa7ac030660 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa7ac0389e0 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa7ac02fb70 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101405804 0 0) 0x7fa7ac0c6de0 con 0x7fa7ac0389e0 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa798004eb0 con 0x7fa7ac0389e0 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791122103 0 0) 0x7fa7ac0c5ec0 con 0x7fa7ac02fb70 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7ac0c6de0 con 0x7fa7ac02fb70 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817744499 0 0) 0x7fa7ac0c6b70 con 0x7fa7ac030660 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7ac0c5ec0 con 0x7fa7ac030660 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2104713522 0 0) 0x7fa7ac0c6de0 con 0x7fa7ac02fb70 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7ac0c6b70 con 0x7fa7ac02fb70 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa7b404fdd0 con 0x7fa7ac02fb70 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): dump: 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:20.423 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.416+0000 7fa7a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3384308565 0 0) 0x7fa7ac0c6b70 con 0x7fa7ac02fb70 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 >> v1:172.21.15.183:6789/0 conn(0x7fa7ac0389e0 legacy=0x7fa7ac0c80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 >> v1:172.21.15.155:6789/0 conn(0x7fa7ac030660 legacy=0x7fa7ac0c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7a5ffb640 1 monclient: found mon.c 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac0c6050 con 0x7fa7ac02fb70 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:20.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7ba06d640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7ba06d640 10 monclient: _renew_subs 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7ba06d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7ba06d640 1 -- 172.21.15.155:0/1274051183 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ac0c58c0 con 0x7fa7ac02fb70 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7ba06d640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7ba06d640 10 monclient: _renew_subs 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7ba06d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7ba06d640 1 -- 172.21.15.155:0/1274051183 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ac1688b0 con 0x7fa7ac02fb70 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.417+0000 7fa7ba06d640 1 librados: init done 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.418+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7b404f400 con 0x7fa7ac02fb70 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.418+0000 7fa7a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.418+0000 7fa7a6ffd640 4 set_mon_vals no callback set 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.418+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa7b4075f10 con 0x7fa7ac02fb70 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.418+0000 7fa7a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:20.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.418+0000 7fa7a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:20.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.418+0000 7fa7a5ffb640 10 monclient: dump: 2024-03-29T11:40:20.426 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:20.426 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:20.426 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:20.426 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:20.426 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:20.426 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:20.426 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:20.426 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:20.426 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:20.426 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:20.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.420+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fa7b4067bf0 con 0x7fa7ac02fb70 2024-03-29T11:40:20.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.421+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fa7b40ff3d0 con 0x7fa7ac02fb70 2024-03-29T11:40:20.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.421+0000 7fa7a5ffb640 10 client.4463.objecter ms_dispatch 0x7fa7b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:20.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.421+0000 7fa7a5ffb640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:20.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.421+0000 7fa7a5ffb640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:20.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.421+0000 7fa7a5ffb640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T11:40:20.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.420+0000 7fa7ba06d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:20.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.421+0000 7fa7ba06d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:20.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.421+0000 7fa7ba06d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:20.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.421+0000 7fa7ba06d640 1 -- 172.21.15.155:0/1274051183 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa794005080 con 0x7fa7ac02fb70 2024-03-29T11:40:20.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.422+0000 7fa7b906b640 10 monclient: get_auth_request con 0x7fa79806dd30 auth_method 0 2024-03-29T11:40:20.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.422+0000 7fa7a5ffb640 10 client.4463.objecter ms_handle_connect 0x7fa79806dd30 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.430+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7b40cacb0 con 0x7fa7ac02fb70 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.430+0000 7fa7a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.430+0000 7fa7a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:20.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.754+0000 7fa7ba06d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:20.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.754+0000 7fa7ba06d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:20.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.754+0000 7fa7ba06d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.754+0000 7fa7ba06d640 1 -- 172.21.15.155:0/1274051183 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa794005330 con 0x7fa7ac02fb70 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.755+0000 7fa7a5ffb640 1 -- 172.21.15.155:0/1274051183 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fa7b40ccbe0 con 0x7fa7ac02fb70 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.755+0000 7fa7a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.755+0000 7fa7a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:20.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.758+0000 7fa7ba06d640 10 librados: watch_flush enter 2024-03-29T11:40:20.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.758+0000 7fa7ba06d640 10 librados: watch_flush exit 2024-03-29T11:40:20.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.758+0000 7fa7ba06d640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T11:40:20.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.758+0000 7fa7ba06d640 10 client.4463.objecter successfully canceled tick 2024-03-29T11:40:20.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.759+0000 7fa7ba06d640 1 -- 172.21.15.155:0/1274051183 >> v1:172.21.15.155:6816/1760814708 conn(0x7fa79806dd30 legacy=0x7fa7980701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:20.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.759+0000 7fa7ba06d640 10 monclient: shutdown 2024-03-29T11:40:20.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.759+0000 7fa7ba06d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:20.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.759+0000 7fa7ba06d640 1 -- 172.21.15.155:0/1274051183 >> v1:172.21.15.155:6790/0 conn(0x7fa7ac02fb70 legacy=0x7fa7ac167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:20.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.759+0000 7fa7ba06d640 1 -- 172.21.15.155:0/1274051183 shutdown_connections 2024-03-29T11:40:20.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.759+0000 7fa7ba06d640 1 -- 172.21.15.155:0/1274051183 >> 172.21.15.155:0/1274051183 conn(0x7fa7ac005c00 msgr2=0x7fa7ac0c43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:20.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.759+0000 7fa7ba06d640 1 -- 172.21.15.155:0/1274051183 shutdown_connections 2024-03-29T11:40:20.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.760+0000 7fa7ba06d640 1 -- 172.21.15.155:0/1274051183 wait complete. 2024-03-29T11:40:20.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:20.760+0000 7fa7ba06d640 1 librados: shutdown 2024-03-29T11:40:20.781 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:27.589 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.741+0000 7f35e0a1b640 10 monclient: get_monmap_and_config 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.741+0000 7f35e0a1b640 10 monclient: build_initial_monmap 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.741+0000 7f35e0a1b640 10 monclient: monmap: 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:40:27.743116+0000 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:40:27.743116+0000 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.743+0000 7f35e0a1b640 1 Processor -- start 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.743+0000 7f35e0a1b640 1 -- start start 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.743+0000 7f35e0a1b640 10 monclient: init 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35e0a1b640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35e0a1b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35e0a1b640 10 monclient(hunting): picked mon.noname-c con 0x7f35d4029f00 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35e0a1b640 10 monclient(hunting): picked mon.noname-a con 0x7f35d402b090 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35e0a1b640 10 monclient(hunting): picked mon.noname-b con 0x7f35d40e2c40 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35e0a1b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d402c0f0 con 0x7f35d402b090 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35e0a1b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d402a430 con 0x7f35d4029f00 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35e0a1b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d402a5c0 con 0x7f35d40e2c40 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35e0a1b640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35e0a1b640 10 monclient(hunting): authenticate will time out at 1209.871161s 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35dbfff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f35d40e2c40 0x7f35d40f0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51764/0 (socket says 172.21.15.155:51764) 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35dbfff640 1 -- 172.21.15.155:0/2417899834 learned_addr learned my addr 172.21.15.155:0/2417899834 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.744+0000 7f35da7fc640 1 -- 172.21.15.155:0/2417899834 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351274923 0 0) 0x7f35d402a5c0 con 0x7f35d40e2c40 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.745+0000 7f35da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.745+0000 7f35da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.745+0000 7f35da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.745+0000 7f35da7fc640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.745+0000 7f35da7fc640 1 -- 172.21.15.155:0/2417899834 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35cc003520 con 0x7f35d40e2c40 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.745+0000 7f35da7fc640 1 -- 172.21.15.155:0/2417899834 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1216076684 0 0) 0x7f35cc003520 con 0x7f35d40e2c40 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.745+0000 7f35da7fc640 1 -- 172.21.15.155:0/2417899834 >> v1:172.21.15.183:6789/0 conn(0x7f35d4029f00 legacy=0x7f35d40e8900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.747+0000 7f35da7fc640 1 -- 172.21.15.155:0/2417899834 >> v1:172.21.15.155:6789/0 conn(0x7f35d402b090 legacy=0x7f35d40e2530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:27.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:27.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 1 monclient: found mon.noname-b 2024-03-29T11:40:27.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:27.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:40:27.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 1 -- 172.21.15.155:0/2417899834 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d4006940 con 0x7f35d40e2c40 2024-03-29T11:40:27.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:27.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 1 -- 172.21.15.155:0/2417899834 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f35c8001690 con 0x7f35d40e2c40 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.749+0000 7f35da7fc640 10 monclient: dump: 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:27.757 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:27.757 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:27.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.750+0000 7f35e0a1b640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:40:27.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.750+0000 7f35e0a1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.751+0000 7f35da7fc640 1 -- 172.21.15.155:0/2417899834 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35c8001860 con 0x7f35d40e2c40 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.751+0000 7f35da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.751+0000 7f35da7fc640 1 -- 172.21.15.155:0/2417899834 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f35c8001cb0 con 0x7f35d40e2c40 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.751+0000 7f35e0a1b640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.751+0000 7f35e0a1b640 4 set_mon_vals no callback set 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.751+0000 7f35da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.751+0000 7f35da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.751+0000 7f35da7fc640 10 monclient: dump: 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:27.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.752+0000 7f35e0a1b640 10 monclient: shutdown 2024-03-29T11:40:27.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.752+0000 7f35e0a1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:27.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.752+0000 7f35e0a1b640 1 -- 172.21.15.155:0/2417899834 >> v1:172.21.15.155:6790/0 conn(0x7f35d40e2c40 legacy=0x7f35d40f0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:27.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.753+0000 7f35e0a1b640 1 -- 172.21.15.155:0/2417899834 shutdown_connections 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.753+0000 7f35e0a1b640 1 -- 172.21.15.155:0/2417899834 >> 172.21.15.155:0/2417899834 conn(0x7f35d40058f0 msgr2=0x7f35d4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.753+0000 7f35e0a1b640 1 -- 172.21.15.155:0/2417899834 shutdown_connections 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.753+0000 7f35e0a1b640 1 -- 172.21.15.155:0/2417899834 wait complete. 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.754+0000 7f35e0a1b640 10 monclient: build_initial_monmap 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.754+0000 7f35e0a1b640 10 monclient: monmap: 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.754+0000 7f35e0a1b640 1 librados: starting msgr at 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.754+0000 7f35e0a1b640 1 librados: starting objecter 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.755+0000 7f35e0a1b640 1 Processor -- start 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.755+0000 7f35e0a1b640 1 -- start start 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.755+0000 7f35e0a1b640 1 librados: setting wanted keys 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.755+0000 7f35e0a1b640 1 librados: calling monclient init 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.755+0000 7f35e0a1b640 10 monclient: init 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.756+0000 7f35e0a1b640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.756+0000 7f35e0a1b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.756+0000 7f35e0a1b640 10 monclient(hunting): picked mon.noname-c con 0x7f35d4029f00 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.756+0000 7f35e0a1b640 10 monclient(hunting): picked mon.noname-b con 0x7f35d402b090 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.756+0000 7f35e0a1b640 10 monclient(hunting): picked mon.noname-a con 0x7f35d40e2c40 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.756+0000 7f35e0a1b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d4031ac0 con 0x7f35d40e2c40 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.756+0000 7f35e0a1b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d40df4a0 con 0x7f35d402b090 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.756+0000 7f35e0a1b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d402e090 con 0x7f35d4029f00 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.756+0000 7f35e0a1b640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.756+0000 7f35e0a1b640 10 monclient(hunting): authenticate will time out at 1209.883296s 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35db7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f35d4029f00 0x7f35d402ea70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55138/0 (socket says 172.21.15.155:55138) 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35db7fe640 1 -- 172.21.15.155:0/1226761688 learned_addr learned my addr 172.21.15.155:0/1226761688 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 10 client.?.objecter ms_handle_connect 0x7f35d4029f00 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170900980 0 0) 0x7f35d402e090 con 0x7f35d4029f00 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c0004eb0 con 0x7f35d4029f00 2024-03-29T11:40:27.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 249092524 0 0) 0x7f35c0004eb0 con 0x7f35d4029f00 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35d402e090 con 0x7f35d4029f00 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f35d0001870 con 0x7f35d4029f00 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 10 monclient(hunting): dump: 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:27.766 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:27.767 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:27.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.758+0000 7f35c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:27.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.759+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 555938314 0 0) 0x7f35d402e090 con 0x7f35d4029f00 2024-03-29T11:40:27.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.759+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 >> v1:172.21.15.183:6789/0 conn(0x7f35d402b090 legacy=0x7f35d40ddb50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:27.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 >> v1:172.21.15.155:6789/0 conn(0x7f35d40e2c40 legacy=0x7f35d418d680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:27.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35c7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:27.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35c7fff640 1 monclient: found mon.c 2024-03-29T11:40:27.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:27.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35c7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:27.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d402e370 con 0x7f35d4029f00 2024-03-29T11:40:27.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35c7fff640 10 monclient: _finish_auth 0 2024-03-29T11:40:27.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35c7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:27.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:27.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35e0a1b640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:40:27.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35e0a1b640 10 monclient: _renew_subs 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35e0a1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35e0a1b640 1 -- 172.21.15.155:0/1226761688 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d402e650 con 0x7f35d4029f00 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35e0a1b640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35e0a1b640 10 monclient: _renew_subs 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35e0a1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35e0a1b640 1 -- 172.21.15.155:0/1226761688 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d4032660 con 0x7f35d4029f00 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.762+0000 7f35e0a1b640 1 librados: init done 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.763+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35d00019d0 con 0x7f35d4029f00 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.763+0000 7f35c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.763+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f35d0004ae0 con 0x7f35d4029f00 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.763+0000 7f35c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.763+0000 7f35c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.763+0000 7f35c7fff640 10 monclient: dump: 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:27.771 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.763+0000 7f35d97fa640 4 set_mon_vals no callback set 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.765+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f35d001aab0 con 0x7f35d4029f00 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.765+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f35d0084b20 con 0x7f35d4029f00 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.766+0000 7f35c7fff640 10 client.4469.objecter ms_dispatch 0x7f35dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.766+0000 7f35c7fff640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.766+0000 7f35c7fff640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:27.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.766+0000 7f35c7fff640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T11:40:27.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.766+0000 7f35e0a1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:27.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.766+0000 7f35e0a1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:27.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.766+0000 7f35e0a1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:27.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.766+0000 7f35e0a1b640 1 -- 172.21.15.155:0/1226761688 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc005080 con 0x7f35d4029f00 2024-03-29T11:40:27.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.767+0000 7f35daffd640 10 monclient: get_auth_request con 0x7f35c006dbe0 auth_method 0 2024-03-29T11:40:27.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.767+0000 7f35c7fff640 10 client.4469.objecter ms_handle_connect 0x7f35c006dbe0 2024-03-29T11:40:27.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.775+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35d0050230 con 0x7f35d4029f00 2024-03-29T11:40:27.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.775+0000 7f35c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:27.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:27.775+0000 7f35c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:28.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.088+0000 7f35e0a1b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:28.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.088+0000 7f35e0a1b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:28.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.088+0000 7f35e0a1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:28.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.088+0000 7f35e0a1b640 1 -- 172.21.15.155:0/1226761688 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35bc005330 con 0x7f35d4029f00 2024-03-29T11:40:28.091 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:40:28.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.089+0000 7f35c7fff640 1 -- 172.21.15.155:0/1226761688 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f35d0052330 con 0x7f35d4029f00 2024-03-29T11:40:28.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.089+0000 7f35c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:28.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.089+0000 7f35c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:28.092 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:28.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.093+0000 7f35e0a1b640 10 librados: watch_flush enter 2024-03-29T11:40:28.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.093+0000 7f35e0a1b640 10 librados: watch_flush exit 2024-03-29T11:40:28.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.093+0000 7f35e0a1b640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T11:40:28.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.093+0000 7f35e0a1b640 10 client.4469.objecter successfully canceled tick 2024-03-29T11:40:28.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.093+0000 7f35e0a1b640 1 -- 172.21.15.155:0/1226761688 >> v1:172.21.15.155:6816/1760814708 conn(0x7f35c006dbe0 legacy=0x7f35c00700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:28.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.093+0000 7f35e0a1b640 10 monclient: shutdown 2024-03-29T11:40:28.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.093+0000 7f35e0a1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:28.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.093+0000 7f35e0a1b640 1 -- 172.21.15.155:0/1226761688 >> v1:172.21.15.155:6790/0 conn(0x7f35d4029f00 legacy=0x7f35d402ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:28.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.094+0000 7f35e0a1b640 1 -- 172.21.15.155:0/1226761688 shutdown_connections 2024-03-29T11:40:28.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.094+0000 7f35e0a1b640 1 -- 172.21.15.155:0/1226761688 >> 172.21.15.155:0/1226761688 conn(0x7f35d40058f0 msgr2=0x7f35d40e47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:28.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.094+0000 7f35e0a1b640 1 -- 172.21.15.155:0/1226761688 shutdown_connections 2024-03-29T11:40:28.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.094+0000 7f35e0a1b640 1 -- 172.21.15.155:0/1226761688 wait complete. 2024-03-29T11:40:28.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:28.094+0000 7f35e0a1b640 1 librados: shutdown 2024-03-29T11:40:28.112 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:34.919 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:35.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.068+0000 7f316d6c2640 10 monclient: get_monmap_and_config 2024-03-29T11:40:35.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.068+0000 7f316d6c2640 10 monclient: build_initial_monmap 2024-03-29T11:40:35.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.068+0000 7f316d6c2640 10 monclient: monmap: 2024-03-29T11:40:35.071 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:35.071 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:35.071 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:40:35.069714+0000 2024-03-29T11:40:35.071 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:40:35.069714+0000 2024-03-29T11:40:35.071 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:35.071 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:35.071 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:35.071 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:40:35.071 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:40:35.071 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:35.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.069+0000 7f316d6c2640 1 Processor -- start 2024-03-29T11:40:35.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.069+0000 7f316d6c2640 1 -- start start 2024-03-29T11:40:35.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.069+0000 7f316d6c2640 10 monclient: init 2024-03-29T11:40:35.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.070+0000 7f316d6c2640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.070+0000 7f316d6c2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.070+0000 7f316d6c2640 10 monclient(hunting): picked mon.noname-b con 0x7f3160038c60 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.070+0000 7f316d6c2640 10 monclient(hunting): picked mon.noname-a con 0x7f31600324b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.070+0000 7f3167fff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f3160038c60 0x7f31600c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:52302/0 (socket says 172.21.15.155:52302) 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.073+0000 7f316cec1640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f31600d11e0 0x7f31600d35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:35739/0 (socket says 172.21.15.155:35739) 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.073+0000 7f316d6c2640 10 monclient(hunting): picked mon.noname-c con 0x7f31600d11e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.073+0000 7f3167fff640 1 -- 172.21.15.155:0/3554931575 learned_addr learned my addr 172.21.15.155:0/3554931575 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.073+0000 7f316d6c2640 1 -- 172.21.15.155:0/3554931575 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31600caf80 con 0x7f31600324b0 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.073+0000 7f316d6c2640 1 -- 172.21.15.155:0/3554931575 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3160039850 con 0x7f31600d11e0 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.073+0000 7f316d6c2640 1 -- 172.21.15.155:0/3554931575 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31600c78d0 con 0x7f3160038c60 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.073+0000 7f316d6c2640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:35.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.073+0000 7f316d6c2640 10 monclient(hunting): authenticate will time out at 1217.200561s 2024-03-29T11:40:35.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 1 -- 172.21.15.155:0/3554931575 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324662739 0 0) 0x7f31600c78d0 con 0x7f3160038c60 2024-03-29T11:40:35.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:35.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 1 -- 172.21.15.155:0/3554931575 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3158003520 con 0x7f3160038c60 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 1 -- 172.21.15.155:0/3554931575 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3044751219 0 0) 0x7f3158003520 con 0x7f3160038c60 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 1 -- 172.21.15.155:0/3554931575 >> v1:172.21.15.183:6789/0 conn(0x7f31600d11e0 legacy=0x7f31600d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 1 -- 172.21.15.155:0/3554931575 >> v1:172.21.15.155:6789/0 conn(0x7f31600324b0 legacy=0x7f31600cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 1 monclient: found mon.noname-b 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 1 -- 172.21.15.155:0/3554931575 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31600c7a60 con 0x7f3160038c60 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:35.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 1 -- 172.21.15.155:0/3554931575 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f315c0015c0 con 0x7f3160038c60 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f316d6c2640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient: dump: 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:35.079 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f316d6c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 1 -- 172.21.15.155:0/3554931575 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f315c001790 con 0x7f3160038c60 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 1 -- 172.21.15.155:0/3554931575 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f315c001ba0 con 0x7f3160038c60 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f3166ffd640 10 monclient: dump: 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:35.080 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:35.081 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:35.081 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:35.081 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:35.081 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:35.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.074+0000 7f316d6c2640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:35.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.075+0000 7f316d6c2640 4 set_mon_vals no callback set 2024-03-29T11:40:35.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.075+0000 7f316d6c2640 10 monclient: shutdown 2024-03-29T11:40:35.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.075+0000 7f316d6c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:35.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.075+0000 7f316d6c2640 1 -- 172.21.15.155:0/3554931575 >> v1:172.21.15.155:6790/0 conn(0x7f3160038c60 legacy=0x7f31600c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.076+0000 7f316d6c2640 1 -- 172.21.15.155:0/3554931575 shutdown_connections 2024-03-29T11:40:35.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.076+0000 7f316d6c2640 1 -- 172.21.15.155:0/3554931575 >> 172.21.15.155:0/3554931575 conn(0x7f3160005c20 msgr2=0x7f3160006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:35.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.076+0000 7f316d6c2640 1 -- 172.21.15.155:0/3554931575 shutdown_connections 2024-03-29T11:40:35.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.076+0000 7f316d6c2640 1 -- 172.21.15.155:0/3554931575 wait complete. 2024-03-29T11:40:35.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.077+0000 7f316d6c2640 10 monclient: build_initial_monmap 2024-03-29T11:40:35.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.077+0000 7f316d6c2640 10 monclient: monmap: 2024-03-29T11:40:35.083 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:35.083 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.077+0000 7f316d6c2640 1 librados: starting msgr at 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.077+0000 7f316d6c2640 1 librados: starting objecter 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.077+0000 7f316d6c2640 1 Processor -- start 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.078+0000 7f316d6c2640 1 -- start start 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.078+0000 7f316d6c2640 1 librados: setting wanted keys 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.078+0000 7f316d6c2640 1 librados: calling monclient init 2024-03-29T11:40:35.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.078+0000 7f316d6c2640 10 monclient: init 2024-03-29T11:40:35.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.078+0000 7f316d6c2640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:35.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.078+0000 7f316d6c2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:35.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f316d6c2640 10 monclient(hunting): picked mon.noname-b con 0x7f31600324b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:35.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f316d6c2640 10 monclient(hunting): picked mon.noname-a con 0x7f3160038c60 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:35.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f3167fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f31600324b0 0x7f3160035650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:35773/0 (socket says 172.21.15.155:35773) 2024-03-29T11:40:35.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f316d6c2640 10 monclient(hunting): picked mon.noname-c con 0x7f31600d11e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:35.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f316d6c2640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31600c31d0 con 0x7f3160038c60 2024-03-29T11:40:35.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f316d6c2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3160036ca0 con 0x7f31600324b0 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f316d6c2640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3160036f10 con 0x7f31600d11e0 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f316d6c2640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f316d6c2640 10 monclient(hunting): authenticate will time out at 1217.209427s 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f316cec1640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f31600d11e0 0x7f3160174150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:56324/0 (socket says 172.21.15.155:56324) 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f316cec1640 1 -- 172.21.15.155:0/1712646950 learned_addr learned my addr 172.21.15.155:0/1712646950 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f3164ff9640 10 client.?.objecter ms_handle_connect 0x7f31600324b0 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f3164ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f3164ff9640 10 client.?.objecter ms_handle_connect 0x7f31600d11e0 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f3164ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.082+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194580171 0 0) 0x7f3160036f10 con 0x7f31600d11e0 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:35.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f314c004eb0 con 0x7f31600d11e0 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2316303650 0 0) 0x7f314c004eb0 con 0x7f31600d11e0 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3160036f10 con 0x7f31600d11e0 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f316804fdd0 con 0x7f31600d11e0 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:35.090 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.083+0000 7f3164ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.084+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767781584 0 0) 0x7f3160036ca0 con 0x7f31600324b0 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.084+0000 7f3164ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.084+0000 7f3164ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.084+0000 7f3164ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.084+0000 7f3164ff9640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.084+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f314c004eb0 con 0x7f31600324b0 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.085+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 336741465 0 0) 0x7f314c004eb0 con 0x7f31600324b0 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.085+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3160036ca0 con 0x7f31600324b0 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.085+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f315c001aa0 con 0x7f31600324b0 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.085+0000 7f3164ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.085+0000 7f3164ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:35.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.085+0000 7f3164ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.085+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4166888142 0 0) 0x7f3160036ca0 con 0x7f31600324b0 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.085+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 >> v1:172.21.15.155:6790/0 conn(0x7f31600d11e0 legacy=0x7f3160174150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.085+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 >> v1:172.21.15.155:6789/0 conn(0x7f3160038c60 legacy=0x7f3160169a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:35.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:35.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 1 monclient: found mon.b 2024-03-29T11:40:35.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:35.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:40:35.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31600348b0 con 0x7f31600324b0 2024-03-29T11:40:35.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:35.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:35.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:35.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f316d6c2640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:40:35.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f316d6c2640 10 monclient: _renew_subs 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f316d6c2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f316d6c2640 1 -- 172.21.15.155:0/1712646950 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3160035d60 con 0x7f31600324b0 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f316d6c2640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f316d6c2640 10 monclient: _renew_subs 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f316d6c2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f316d6c2640 1 -- 172.21.15.155:0/1712646950 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f316016a350 con 0x7f31600324b0 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f316d6c2640 1 librados: init done 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f315c001c00 con 0x7f31600324b0 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f315c004c70 con 0x7f31600324b0 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3164ff9640 10 monclient: dump: 2024-03-29T11:40:35.096 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.087+0000 7f3165ffb640 4 set_mon_vals no callback set 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.089+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f315c01ac40 con 0x7f31600324b0 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.090+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f315c084af0 con 0x7f31600324b0 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.090+0000 7f31677fe640 10 monclient: get_auth_request con 0x7f314c06dd70 auth_method 0 2024-03-29T11:40:35.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.090+0000 7f3164ff9640 10 client.4423.objecter ms_dispatch 0x7f3168000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:35.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.090+0000 7f3164ff9640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:35.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.090+0000 7f3164ff9640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:35.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.091+0000 7f3164ff9640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:40:35.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.091+0000 7f3164ff9640 10 client.4423.objecter ms_handle_connect 0x7f314c06dd70 2024-03-29T11:40:35.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.092+0000 7f316d6c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:35.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.092+0000 7f316d6c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:35.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.092+0000 7f316d6c2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:40:35.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.092+0000 7f316d6c2640 1 -- 172.21.15.155:0/1712646950 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3160007b70 con 0x7f31600324b0 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.100+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f315c0503d0 con 0x7f31600324b0 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.100+0000 7f3164ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.100+0000 7f3164ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:35.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.427+0000 7f316d6c2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:35.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.427+0000 7f316d6c2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:35.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.427+0000 7f316d6c2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:40:35.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.427+0000 7f316d6c2640 1 -- 172.21.15.155:0/1712646950 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3160004cd0 con 0x7f31600324b0 2024-03-29T11:40:35.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.429+0000 7f3164ff9640 1 -- 172.21.15.155:0/1712646950 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f315c052300 con 0x7f31600324b0 2024-03-29T11:40:35.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.429+0000 7f3164ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:35.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.429+0000 7f3164ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:35.432 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:40:35.432 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:35.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.434+0000 7f316d6c2640 10 librados: watch_flush enter 2024-03-29T11:40:35.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.434+0000 7f316d6c2640 10 librados: watch_flush exit 2024-03-29T11:40:35.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.434+0000 7f316d6c2640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:40:35.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.434+0000 7f316d6c2640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:40:35.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.434+0000 7f316d6c2640 1 -- 172.21.15.155:0/1712646950 >> v1:172.21.15.155:6816/1760814708 conn(0x7f314c06dd70 legacy=0x7f314c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.434+0000 7f316d6c2640 10 monclient: shutdown 2024-03-29T11:40:35.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.434+0000 7f316d6c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:35.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.434+0000 7f316d6c2640 1 -- 172.21.15.155:0/1712646950 >> v1:172.21.15.183:6789/0 conn(0x7f31600324b0 legacy=0x7f3160035650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.434+0000 7f316d6c2640 1 -- 172.21.15.155:0/1712646950 shutdown_connections 2024-03-29T11:40:35.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.434+0000 7f316d6c2640 1 -- 172.21.15.155:0/1712646950 >> 172.21.15.155:0/1712646950 conn(0x7f3160005c20 msgr2=0x7f31600043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:35.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.435+0000 7f316d6c2640 1 -- 172.21.15.155:0/1712646950 shutdown_connections 2024-03-29T11:40:35.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.435+0000 7f316d6c2640 1 -- 172.21.15.155:0/1712646950 wait complete. 2024-03-29T11:40:35.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:35.435+0000 7f316d6c2640 1 librados: shutdown 2024-03-29T11:40:35.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:42.261 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:42.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.418+0000 7f53f7c4a640 10 monclient: get_monmap_and_config 2024-03-29T11:40:42.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.418+0000 7f53f7c4a640 10 monclient: build_initial_monmap 2024-03-29T11:40:42.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.418+0000 7f53f7c4a640 10 monclient: monmap: 2024-03-29T11:40:42.420 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:42.420 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:42.420 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:40:42.418773+0000 2024-03-29T11:40:42.420 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:40:42.418773+0000 2024-03-29T11:40:42.420 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:42.420 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:42.420 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:42.420 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:40:42.421 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:40:42.421 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:42.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.419+0000 7f53f7c4a640 1 Processor -- start 2024-03-29T11:40:42.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.419+0000 7f53f7c4a640 1 -- start start 2024-03-29T11:40:42.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.419+0000 7f53f7c4a640 10 monclient: init 2024-03-29T11:40:42.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.420+0000 7f53f7c4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:42.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.420+0000 7f53f7c4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:42.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.420+0000 7f53f7c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f53f01422b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:42.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.420+0000 7f53f7c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f53f0142690 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:42.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.420+0000 7f53f7c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f53f00b17d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:42.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.420+0000 7f53f7c4a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f0142c80 con 0x7f53f01422b0 2024-03-29T11:40:42.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.420+0000 7f53f7c4a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f00aae40 con 0x7f53f0142690 2024-03-29T11:40:42.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.420+0000 7f53f7c4a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f013fbd0 con 0x7f53f00b17d0 2024-03-29T11:40:42.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.420+0000 7f53f7c4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:42.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.420+0000 7f53f7c4a640 10 monclient(hunting): authenticate will time out at 1224.546309s 2024-03-29T11:40:42.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.423+0000 7f53f59e6640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f53f01422b0 0x7f53f013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:35253/0 (socket says 172.21.15.155:35253) 2024-03-29T11:40:42.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f59e6640 1 -- 172.21.15.155:0/2908023619 learned_addr learned my addr 172.21.15.155:0/2908023619 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:42.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.424+0000 7f53f51e5640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f53f0142690 0x7f53f00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:55587/0 (socket says 172.21.15.155:55587) 2024-03-29T11:40:42.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.424+0000 7f53f61e7640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f53f00b17d0 0x7f53f00b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:44208/0 (socket says 172.21.15.155:44208) 2024-03-29T11:40:42.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 1 -- 172.21.15.155:0/2908023619 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899882154 0 0) 0x7f53f0142c80 con 0x7f53f01422b0 2024-03-29T11:40:42.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 10 monclient(hunting): _negotiate 2024-03-29T11:40:42.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:42.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:42.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:40:42.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 1 -- 172.21.15.155:0/2908023619 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53dc003520 con 0x7f53f01422b0 2024-03-29T11:40:42.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 1 -- 172.21.15.155:0/2908023619 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162108620 0 0) 0x7f53f00aae40 con 0x7f53f0142690 2024-03-29T11:40:42.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 10 monclient(hunting): _negotiate 2024-03-29T11:40:42.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:42.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:42.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:40:42.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.425+0000 7f53f49e4640 1 -- 172.21.15.155:0/2908023619 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f0142c80 con 0x7f53f0142690 2024-03-29T11:40:42.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 1 -- 172.21.15.155:0/2908023619 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 871499993 0 0) 0x7f53f0142c80 con 0x7f53f0142690 2024-03-29T11:40:42.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 1 -- 172.21.15.155:0/2908023619 >> v1:172.21.15.155:6790/0 conn(0x7f53f00b17d0 legacy=0x7f53f00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:42.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 1 -- 172.21.15.155:0/2908023619 >> v1:172.21.15.155:6789/0 conn(0x7f53f01422b0 legacy=0x7f53f013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:42.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 10 monclient: _finish_hunting 0 2024-03-29T11:40:42.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 1 monclient: found mon.noname-c 2024-03-29T11:40:42.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:42.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 1 -- 172.21.15.155:0/2908023619 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f0151940 con 0x7f53f0142690 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 10 monclient: _finish_auth 0 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 10 monclient: _check_auth_tickets 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 1 -- 172.21.15.155:0/2908023619 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f53e40015a0 con 0x7f53f0142690 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 10 monclient: dump: 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:42.432 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f7c4a640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f7c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 1 -- 172.21.15.155:0/2908023619 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53e4001700 con 0x7f53f0142690 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 1 -- 172.21.15.155:0/2908023619 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f53e4001a90 con 0x7f53f0142690 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f7c4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f7c4a640 4 set_mon_vals no callback set 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:42.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.426+0000 7f53f49e4640 10 monclient: dump: 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.427+0000 7f53f7c4a640 10 monclient: shutdown 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.427+0000 7f53f7c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.427+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2908023619 >> v1:172.21.15.183:6789/0 conn(0x7f53f0142690 legacy=0x7f53f00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:42.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.427+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2908023619 shutdown_connections 2024-03-29T11:40:42.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.427+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2908023619 >> 172.21.15.155:0/2908023619 conn(0x7f53f0066a00 msgr2=0x7f53f0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:42.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.427+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2908023619 shutdown_connections 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.427+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2908023619 wait complete. 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.428+0000 7f53f7c4a640 10 monclient: build_initial_monmap 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.428+0000 7f53f7c4a640 10 monclient: monmap: 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:40:42.440 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.429+0000 7f53f7c4a640 1 librados: starting msgr at 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.429+0000 7f53f7c4a640 1 librados: starting objecter 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.429+0000 7f53f7c4a640 1 Processor -- start 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.429+0000 7f53f7c4a640 1 -- start start 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.429+0000 7f53f7c4a640 1 librados: setting wanted keys 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.429+0000 7f53f7c4a640 1 librados: calling monclient init 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.429+0000 7f53f7c4a640 10 monclient: init 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.430+0000 7f53f7c4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.430+0000 7f53f7c4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.430+0000 7f53f7c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f53f00b17d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.430+0000 7f53f7c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f53f01422b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.430+0000 7f53f7c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f53f0142690 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.430+0000 7f53f7c4a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f00acf40 con 0x7f53f0142690 2024-03-29T11:40:42.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.430+0000 7f53f7c4a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f00ad1b0 con 0x7f53f00b17d0 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.430+0000 7f53f7c4a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f00abfb0 con 0x7f53f01422b0 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.430+0000 7f53f7c4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.430+0000 7f53f7c4a640 10 monclient(hunting): authenticate will time out at 1224.556407s 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.431+0000 7f53f51e5640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f53f01422b0 0x7f53f00b0930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:45232/0 (socket says 172.21.15.155:45232) 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.431+0000 7f53f51e5640 1 -- 172.21.15.155:0/2686892279 learned_addr learned my addr 172.21.15.155:0/2686892279 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.431+0000 7f53e27fc640 10 client.?.objecter ms_handle_connect 0x7f53f01422b0 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.431+0000 7f53e27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.432+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444144073 0 0) 0x7f53f00abfb0 con 0x7f53f01422b0 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.432+0000 7f53e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.432+0000 7f53e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.432+0000 7f53e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.432+0000 7f53e27fc640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.432+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53dc0049a0 con 0x7f53f01422b0 2024-03-29T11:40:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.433+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 402669814 0 0) 0x7f53dc0049a0 con 0x7f53f01422b0 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.433+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53f00abfb0 con 0x7f53f01422b0 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.433+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f53e40013c0 con 0x7f53f01422b0 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.433+0000 7f53e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.433+0000 7f53e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.433+0000 7f53e27fc640 10 monclient(hunting): dump: 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:42.443 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.433+0000 7f53e27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.433+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3377056851 0 0) 0x7f53f00abfb0 con 0x7f53f01422b0 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.433+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 >> v1:172.21.15.183:6789/0 conn(0x7f53f00b17d0 legacy=0x7f53f012e3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 >> v1:172.21.15.155:6789/0 conn(0x7f53f0142690 legacy=0x7f53f00875b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53e27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53e27fc640 1 monclient: found mon.c 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53e27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f012eb00 con 0x7f53f01422b0 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53e27fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53e27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:42.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53f7c4a640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53f7c4a640 10 monclient: _renew_subs 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53f7c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2686892279 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f0132210 con 0x7f53f01422b0 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53f7c4a640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53f7c4a640 10 monclient: _renew_subs 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53f7c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2686892279 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f0151160 con 0x7f53f01422b0 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53f7c4a640 1 librados: init done 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53e27fc640 10 client.4481.objecter ms_handle_connect 0x7f53f0142690 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.434+0000 7f53e27fc640 10 client.4481.objecter resend_mon_ops 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.436+0000 7f53f7c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.436+0000 7f53f7c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.436+0000 7f53f7c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.436+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2686892279 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f0075f10 con 0x7f53f01422b0 2024-03-29T11:40:42.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.436+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53e4004a90 con 0x7f53f01422b0 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.436+0000 7f53e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.436+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f53e4004ea0 con 0x7f53f01422b0 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.436+0000 7f53e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.436+0000 7f53e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.436+0000 7f53e27fc640 10 monclient: dump: 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:42.446 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:42.447 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:42.447 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:42.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.436+0000 7f53e37fe640 4 set_mon_vals no callback set 2024-03-29T11:40:42.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.437+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f53e40050c0 con 0x7f53f01422b0 2024-03-29T11:40:42.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.438+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f53e4084be0 con 0x7f53f01422b0 2024-03-29T11:40:42.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.438+0000 7f53e27fc640 10 client.4481.objecter ms_dispatch 0x7f53f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:42.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.438+0000 7f53e27fc640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:42.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.438+0000 7f53e27fc640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:42.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.438+0000 7f53e27fc640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:40:42.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.438+0000 7f53f59e6640 10 monclient: get_auth_request con 0x7f53dc06df70 auth_method 0 2024-03-29T11:40:42.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.443+0000 7f53e27fc640 10 client.4481.objecter ms_handle_connect 0x7f53dc06df70 2024-03-29T11:40:42.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.446+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53e40504c0 con 0x7f53f01422b0 2024-03-29T11:40:42.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.446+0000 7f53e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:42.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.446+0000 7f53e27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:42.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.774+0000 7f53f7c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:42.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.774+0000 7f53f7c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:42.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.774+0000 7f53f7c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:42.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.774+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2686892279 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53f00b2270 con 0x7f53f01422b0 2024-03-29T11:40:42.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.775+0000 7f53e27fc640 1 -- 172.21.15.155:0/2686892279 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f53e40523f0 con 0x7f53f01422b0 2024-03-29T11:40:42.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.775+0000 7f53e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:42.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.775+0000 7f53e27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:42.778 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:40:42.778 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:42.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.781+0000 7f53f7c4a640 10 librados: watch_flush enter 2024-03-29T11:40:42.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.781+0000 7f53f7c4a640 10 librados: watch_flush exit 2024-03-29T11:40:42.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.781+0000 7f53f7c4a640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:40:42.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.781+0000 7f53f7c4a640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:40:42.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.782+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2686892279 >> v1:172.21.15.155:6816/1760814708 conn(0x7f53dc06df70 legacy=0x7f53dc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:42.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.782+0000 7f53f7c4a640 10 monclient: shutdown 2024-03-29T11:40:42.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.782+0000 7f53f7c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:42.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.782+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2686892279 >> v1:172.21.15.155:6790/0 conn(0x7f53f01422b0 legacy=0x7f53f00b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:42.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.782+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2686892279 shutdown_connections 2024-03-29T11:40:42.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.782+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2686892279 >> 172.21.15.155:0/2686892279 conn(0x7f53f0066a00 msgr2=0x7f53f013d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:42.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.782+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2686892279 shutdown_connections 2024-03-29T11:40:42.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.782+0000 7f53f7c4a640 1 -- 172.21.15.155:0/2686892279 wait complete. 2024-03-29T11:40:42.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:42.782+0000 7f53f7c4a640 1 librados: shutdown 2024-03-29T11:40:42.802 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:49.609 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:49.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.764+0000 7faa65284640 10 monclient: get_monmap_and_config 2024-03-29T11:40:49.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.764+0000 7faa65284640 10 monclient: build_initial_monmap 2024-03-29T11:40:49.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.764+0000 7faa65284640 10 monclient: monmap: 2024-03-29T11:40:49.766 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:49.767 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:49.767 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:40:49.765291+0000 2024-03-29T11:40:49.767 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:40:49.765291+0000 2024-03-29T11:40:49.767 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:49.767 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:49.767 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:49.767 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:40:49.767 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:40:49.767 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:49.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.766+0000 7faa65284640 1 Processor -- start 2024-03-29T11:40:49.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.766+0000 7faa65284640 1 -- start start 2024-03-29T11:40:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.766+0000 7faa65284640 10 monclient: init 2024-03-29T11:40:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa65284640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa65284640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa65284640 10 monclient(hunting): picked mon.noname-b con 0x7faa600b4820 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa65284640 10 monclient(hunting): picked mon.noname-c con 0x7faa600a9ce0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa65284640 10 monclient(hunting): picked mon.noname-a con 0x7faa600b3cd0 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa65284640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa600b4180 con 0x7faa600b3cd0 2024-03-29T11:40:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa65284640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa600aaf60 con 0x7faa600a9ce0 2024-03-29T11:40:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa65284640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6013c200 con 0x7faa600b4820 2024-03-29T11:40:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa65284640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa65284640 10 monclient(hunting): authenticate will time out at 1231.893194s 2024-03-29T11:40:49.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa5ffff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7faa600b4820 0x7faa6013baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:53298/0 (socket says 172.21.15.155:53298) 2024-03-29T11:40:49.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa5ffff640 1 -- 172.21.15.155:0/3647964939 learned_addr learned my addr 172.21.15.155:0/3647964939 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:49.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343658531 0 0) 0x7faa600aaf60 con 0x7faa600a9ce0 2024-03-29T11:40:49.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa5effd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:49.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.767+0000 7faa5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:49.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:49.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:40:49.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa4c003520 con 0x7faa600a9ce0 2024-03-29T11:40:49.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254470944 0 0) 0x7faa6013c200 con 0x7faa600b4820 2024-03-29T11:40:49.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:49.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:49.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:49.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:40:49.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa600aaf60 con 0x7faa600b4820 2024-03-29T11:40:49.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738791189 0 0) 0x7faa580013c0 con 0x7faa600b3cd0 2024-03-29T11:40:49.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:49.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:49.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:49.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:40:49.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa6013c200 con 0x7faa600b3cd0 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 435507358 0 0) 0x7faa6013c200 con 0x7faa600b3cd0 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 >> v1:172.21.15.155:6790/0 conn(0x7faa600b4820 legacy=0x7faa6013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 >> v1:172.21.15.183:6789/0 conn(0x7faa600a9ce0 legacy=0x7faa60145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 monclient: found mon.noname-a 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa60145e80 con 0x7faa600b3cd0 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient: _finish_auth 0 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7faa58001850 con 0x7faa600b3cd0 2024-03-29T11:40:49.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient: dump: 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:49.774 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa65284640 5 monclient: authenticate success, global_id 4440 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa65284640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa580019b0 con 0x7faa600b3cd0 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa65284640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa65284640 4 set_mon_vals no callback set 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 1 -- 172.21.15.155:0/3647964939 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7faa58001e00 con 0x7faa600b3cd0 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.768+0000 7faa5effd640 10 monclient: dump: 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:49.775 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.769+0000 7faa65284640 10 monclient: shutdown 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.769+0000 7faa65284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.769+0000 7faa65284640 1 -- 172.21.15.155:0/3647964939 >> v1:172.21.15.155:6789/0 conn(0x7faa600b3cd0 legacy=0x7faa6014d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.770+0000 7faa65284640 1 -- 172.21.15.155:0/3647964939 shutdown_connections 2024-03-29T11:40:49.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.770+0000 7faa65284640 1 -- 172.21.15.155:0/3647964939 >> 172.21.15.155:0/3647964939 conn(0x7faa6006d070 msgr2=0x7faa60066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:49.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.770+0000 7faa65284640 1 -- 172.21.15.155:0/3647964939 shutdown_connections 2024-03-29T11:40:49.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.770+0000 7faa65284640 1 -- 172.21.15.155:0/3647964939 wait complete. 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.771+0000 7faa65284640 10 monclient: build_initial_monmap 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.771+0000 7faa65284640 10 monclient: monmap: 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:40:49.783 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.772+0000 7faa65284640 1 librados: starting msgr at 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.772+0000 7faa65284640 1 librados: starting objecter 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.772+0000 7faa65284640 1 Processor -- start 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.772+0000 7faa65284640 1 -- start start 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.772+0000 7faa65284640 1 librados: setting wanted keys 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.772+0000 7faa65284640 1 librados: calling monclient init 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.772+0000 7faa65284640 10 monclient: init 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.773+0000 7faa65284640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.773+0000 7faa65284640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.774+0000 7faa65284640 10 monclient(hunting): picked mon.noname-b con 0x7faa600a9ce0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.774+0000 7faa65284640 10 monclient(hunting): picked mon.noname-a con 0x7faa600b3cd0 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.774+0000 7faa65284640 10 monclient(hunting): picked mon.noname-c con 0x7faa600b4820 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:49.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.774+0000 7faa5f7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7faa600b3cd0 0x7faa6013a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:38842/0 (socket says 172.21.15.155:38842) 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.774+0000 7faa5f7fe640 1 -- 172.21.15.155:0/3576664 learned_addr learned my addr 172.21.15.155:0/3576664 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.774+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6006fdc0 con 0x7faa600b3cd0 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.774+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa60071a10 con 0x7faa600a9ce0 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.774+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6006fff0 con 0x7faa600b4820 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa65284640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 client.?.objecter ms_handle_connect 0x7faa600b3cd0 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245383432 0 0) 0x7faa6006fdc0 con 0x7faa600b3cd0 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa65284640 10 monclient(hunting): authenticate will time out at 1231.901097s 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:49.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa40004eb0 con 0x7faa600b3cd0 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 client.?.objecter ms_handle_connect 0x7faa600b4820 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406864622 0 0) 0x7faa6006fff0 con 0x7faa600b4820 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa6006fdc0 con 0x7faa600b4820 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.775+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070209267 0 0) 0x7faa40004eb0 con 0x7faa600b3cd0 2024-03-29T11:40:49.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa6006fff0 con 0x7faa600b3cd0 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7faa48001a30 con 0x7faa600b3cd0 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 10 monclient(hunting): dump: 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:49.787 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763464763 0 0) 0x7faa6006fdc0 con 0x7faa600b4820 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa40004eb0 con 0x7faa600b4820 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7faa58001cc0 con 0x7faa600b4820 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 10 monclient(hunting): dump: 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:49.788 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:49.789 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:49.789 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:49.789 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:49.789 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:49.789 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:49.789 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:49.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2296163548 0 0) 0x7faa40004eb0 con 0x7faa600b4820 2024-03-29T11:40:49.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 >> v1:172.21.15.183:6789/0 conn(0x7faa600a9ce0 legacy=0x7faa601de8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 >> v1:172.21.15.155:6789/0 conn(0x7faa600b3cd0 legacy=0x7faa6013a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:49.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 1 monclient: found mon.c 2024-03-29T11:40:49.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:49.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:49.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa601defb0 con 0x7faa600b4820 2024-03-29T11:40:49.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:49.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:49.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:49.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa65284640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:40:49.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa65284640 10 monclient: _renew_subs 2024-03-29T11:40:49.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa65284640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa60071e30 con 0x7faa600b4820 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.776+0000 7faa65284640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.777+0000 7faa65284640 10 monclient: _renew_subs 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.777+0000 7faa65284640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.777+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6006f3b0 con 0x7faa600b4820 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.777+0000 7faa65284640 1 librados: init done 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.777+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa58001e20 con 0x7faa600b4820 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.777+0000 7faa5cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.777+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7faa58003dc0 con 0x7faa600b4820 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.777+0000 7faa5e7fc640 4 set_mon_vals no callback set 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.778+0000 7faa5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.778+0000 7faa5cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:49.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.778+0000 7faa5cff9640 10 monclient: dump: 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.778+0000 7faa65284640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.779+0000 7faa65284640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:49.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.779+0000 7faa65284640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.779+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa44005080 con 0x7faa600b4820 2024-03-29T11:40:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.780+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7faa58019d90 con 0x7faa600b4820 2024-03-29T11:40:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.780+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7faa58083b90 con 0x7faa600b4820 2024-03-29T11:40:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.781+0000 7faa5cff9640 10 client.4487.objecter ms_dispatch 0x7faa60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.781+0000 7faa5ffff640 10 monclient: get_auth_request con 0x7faa4006dec0 auth_method 0 2024-03-29T11:40:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.781+0000 7faa5cff9640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.781+0000 7faa5cff9640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.781+0000 7faa5cff9640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T11:40:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.781+0000 7faa5cff9640 10 client.4487.objecter ms_handle_connect 0x7faa4006dec0 2024-03-29T11:40:49.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.789+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa5804f470 con 0x7faa600b4820 2024-03-29T11:40:49.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.789+0000 7faa5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:49.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:49.789+0000 7faa5cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:50.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.111+0000 7faa65284640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:50.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.111+0000 7faa65284640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:50.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.111+0000 7faa65284640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:50.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.111+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa44005330 con 0x7faa600b4820 2024-03-29T11:40:50.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.112+0000 7faa5cff9640 1 -- 172.21.15.155:0/3576664 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7faa580513a0 con 0x7faa600b4820 2024-03-29T11:40:50.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.112+0000 7faa5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:50.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.112+0000 7faa5cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:50.115 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:40:50.115 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:50.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 10 librados: watch_flush enter 2024-03-29T11:40:50.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 10 librados: watch_flush exit 2024-03-29T11:40:50.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T11:40:50.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 10 client.4487.objecter successfully canceled tick 2024-03-29T11:40:50.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 >> v1:172.21.15.155:6816/1760814708 conn(0x7faa4006dec0 legacy=0x7faa40070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:50.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 10 monclient: shutdown 2024-03-29T11:40:50.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:50.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 >> v1:172.21.15.155:6790/0 conn(0x7faa600b4820 legacy=0x7faa601e6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:50.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 shutdown_connections 2024-03-29T11:40:50.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 >> 172.21.15.155:0/3576664 conn(0x7faa6006d070 msgr2=0x7faa600b5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:50.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 shutdown_connections 2024-03-29T11:40:50.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.115+0000 7faa65284640 1 -- 172.21.15.155:0/3576664 wait complete. 2024-03-29T11:40:50.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:50.116+0000 7faa65284640 1 librados: shutdown 2024-03-29T11:40:50.132 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:56.938 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:57.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.088+0000 7f4b83831640 10 monclient: get_monmap_and_config 2024-03-29T11:40:57.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.088+0000 7f4b83831640 10 monclient: build_initial_monmap 2024-03-29T11:40:57.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.088+0000 7f4b83831640 10 monclient: monmap: 2024-03-29T11:40:57.092 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:57.092 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:57.092 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:40:57.090313+0000 2024-03-29T11:40:57.092 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:40:57.090313+0000 2024-03-29T11:40:57.092 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:57.092 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:57.092 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:57.092 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:40:57.092 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:40:57.092 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:57.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.090+0000 7f4b83831640 1 Processor -- start 2024-03-29T11:40:57.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.090+0000 7f4b83831640 1 -- start start 2024-03-29T11:40:57.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.090+0000 7f4b83831640 10 monclient: init 2024-03-29T11:40:57.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b83831640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:57.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b83831640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:57.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b83831640 10 monclient(hunting): picked mon.noname-a con 0x7f4b7c071de0 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b83831640 10 monclient(hunting): picked mon.noname-c con 0x7f4b7c0721c0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b83831640 10 monclient(hunting): picked mon.noname-b con 0x7f4b7c145ae0 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b83831640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c145650 con 0x7f4b7c071de0 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b83831640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c0ae260 con 0x7f4b7c0721c0 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b83831640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c1420d0 con 0x7f4b7c145ae0 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b83831640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b83831640 10 monclient(hunting): authenticate will time out at 1239.218287s 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b81dce640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f4b7c145ae0 0x7f4b7c145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:59346/0 (socket says 172.21.15.155:59346) 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.091+0000 7f4b81dce640 1 -- 172.21.15.155:0/3682471527 learned_addr learned my addr 172.21.15.155:0/3682471527 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384637384 0 0) 0x7f4b7c1420d0 con 0x7f4b7c145ae0 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:57.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b68003520 con 0x7f4b7c145ae0 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241565875 0 0) 0x7f4b7c0ae260 con 0x7f4b7c0721c0 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b7c1420d0 con 0x7f4b7c0721c0 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890164662 0 0) 0x7f4b7c145650 con 0x7f4b7c071de0 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:40:57.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b7c0ae260 con 0x7f4b7c071de0 2024-03-29T11:40:57.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3367360470 0 0) 0x7f4b7c1420d0 con 0x7f4b7c0721c0 2024-03-29T11:40:57.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 >> v1:172.21.15.155:6790/0 conn(0x7f4b7c145ae0 legacy=0x7f4b7c145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 >> v1:172.21.15.155:6789/0 conn(0x7f4b7c071de0 legacy=0x7f4b7c0aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:57.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 monclient: found mon.noname-c 2024-03-29T11:40:57.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:57.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:40:57.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c0b0590 con 0x7f4b7c0721c0 2024-03-29T11:40:57.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient: _finish_auth 0 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4b6c001670 con 0x7f4b7c0721c0 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b73fff640 10 monclient: dump: 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:57.099 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b83831640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.092+0000 7f4b83831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.093+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b6c001890 con 0x7f4b7c0721c0 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.093+0000 7f4b73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.093+0000 7f4b83831640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.093+0000 7f4b83831640 4 set_mon_vals no callback set 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.093+0000 7f4b73fff640 1 -- 172.21.15.155:0/3682471527 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4b6c001ba0 con 0x7f4b7c0721c0 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.093+0000 7f4b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.093+0000 7f4b73fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.093+0000 7f4b73fff640 10 monclient: dump: 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:57.100 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.093+0000 7f4b83831640 10 monclient: shutdown 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.093+0000 7f4b83831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.093+0000 7f4b83831640 1 -- 172.21.15.155:0/3682471527 >> v1:172.21.15.183:6789/0 conn(0x7f4b7c0721c0 legacy=0x7f4b7c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.094+0000 7f4b83831640 1 -- 172.21.15.155:0/3682471527 shutdown_connections 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.094+0000 7f4b83831640 1 -- 172.21.15.155:0/3682471527 >> 172.21.15.155:0/3682471527 conn(0x7f4b7c06b660 msgr2=0x7f4b7c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.094+0000 7f4b83831640 1 -- 172.21.15.155:0/3682471527 shutdown_connections 2024-03-29T11:40:57.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.094+0000 7f4b83831640 1 -- 172.21.15.155:0/3682471527 wait complete. 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.095+0000 7f4b83831640 10 monclient: build_initial_monmap 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.095+0000 7f4b83831640 10 monclient: monmap: 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:40:57.102 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:57.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.095+0000 7f4b83831640 1 librados: starting msgr at 2024-03-29T11:40:57.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.095+0000 7f4b83831640 1 librados: starting objecter 2024-03-29T11:40:57.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.095+0000 7f4b83831640 1 Processor -- start 2024-03-29T11:40:57.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.096+0000 7f4b83831640 1 -- start start 2024-03-29T11:40:57.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.096+0000 7f4b83831640 1 librados: setting wanted keys 2024-03-29T11:40:57.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.096+0000 7f4b83831640 1 librados: calling monclient init 2024-03-29T11:40:57.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.096+0000 7f4b83831640 10 monclient: init 2024-03-29T11:40:57.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.096+0000 7f4b83831640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:57.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.096+0000 7f4b83831640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:57.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.096+0000 7f4b83831640 10 monclient(hunting): picked mon.noname-b con 0x7f4b7c071de0 addr v1:172.21.15.183:6789/0 2024-03-29T11:40:57.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b83831640 10 monclient(hunting): picked mon.noname-a con 0x7f4b7c145ae0 addr v1:172.21.15.155:6789/0 2024-03-29T11:40:57.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b815cd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f4b7c071de0 0x7f4b7c0acd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:49548/0 (socket says 172.21.15.155:49548) 2024-03-29T11:40:57.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b81dce640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f4b7c1363d0 0x7f4b7c1e5d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:59580/0 (socket says 172.21.15.155:59580) 2024-03-29T11:40:57.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b815cd640 1 -- 172.21.15.155:0/3410265071 learned_addr learned my addr 172.21.15.155:0/3410265071 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:40:57.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b83831640 10 monclient(hunting): picked mon.noname-c con 0x7f4b7c1363d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:40:57.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c0ac130 con 0x7f4b7c145ae0 2024-03-29T11:40:57.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c0ac810 con 0x7f4b7c071de0 2024-03-29T11:40:57.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c0ac4c0 con 0x7f4b7c1363d0 2024-03-29T11:40:57.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b83831640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:57.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b83831640 10 monclient(hunting): authenticate will time out at 1239.224535s 2024-03-29T11:40:57.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b71ffb640 10 client.?.objecter ms_handle_connect 0x7f4b7c1363d0 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b71ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b71ffb640 10 client.?.objecter ms_handle_connect 0x7f4b7c071de0 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b71ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b71ffb640 10 client.?.objecter ms_handle_connect 0x7f4b7c145ae0 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b71ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104760255 0 0) 0x7f4b7c0ac810 con 0x7f4b7c071de0 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b71ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.097+0000 7f4b71ffb640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b60004eb0 con 0x7f4b7c071de0 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150483660 0 0) 0x7f4b7c0ac4c0 con 0x7f4b7c1363d0 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:57.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b7c0ac810 con 0x7f4b7c1363d0 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350546093 0 0) 0x7f4b7c0ac130 con 0x7f4b7c145ae0 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b7c0ac4c0 con 0x7f4b7c145ae0 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1648740567 0 0) 0x7f4b7c0ac810 con 0x7f4b7c1363d0 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b7c0ac130 con 0x7f4b7c1363d0 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4b74001c50 con 0x7f4b7c1363d0 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:57.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): dump: 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 199318032 0 0) 0x7f4b7c0ac4c0 con 0x7f4b7c145ae0 2024-03-29T11:40:57.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b7c0ac810 con 0x7f4b7c145ae0 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4b6c001530 con 0x7f4b7c145ae0 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.098+0000 7f4b71ffb640 10 monclient(hunting): dump: 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:57.114 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3862962951 0 0) 0x7f4b60004eb0 con 0x7f4b7c071de0 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b7c0ac4c0 con 0x7f4b7c071de0 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4b780018f0 con 0x7f4b7c071de0 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 10 monclient(hunting): dump: 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:57.115 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3500675379 0 0) 0x7f4b7c0ac130 con 0x7f4b7c1363d0 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 >> v1:172.21.15.183:6789/0 conn(0x7f4b7c071de0 legacy=0x7f4b7c0acd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 >> v1:172.21.15.155:6789/0 conn(0x7f4b7c145ae0 legacy=0x7f4b7c136f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 1 monclient: found mon.c 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c137610 con 0x7f4b7c1363d0 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:57.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b83831640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b83831640 10 monclient: _renew_subs 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b83831640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7c1378c0 con 0x7f4b7c1363d0 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b83831640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b83831640 10 monclient: _renew_subs 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b83831640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7c1e7610 con 0x7f4b7c1363d0 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.099+0000 7f4b83831640 1 librados: init done 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.101+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b74004b60 con 0x7f4b7c1363d0 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.101+0000 7f4b71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.101+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4b74004e70 con 0x7f4b7c1363d0 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.101+0000 7f4b71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:57.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.101+0000 7f4b71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.101+0000 7f4b71ffb640 10 monclient: dump: 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.101+0000 7f4b737fe640 4 set_mon_vals no callback set 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.101+0000 7f4b83831640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:57.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.102+0000 7f4b83831640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:57.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.102+0000 7f4b83831640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:57.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.102+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5c005080 con 0x7f4b7c1363d0 2024-03-29T11:40:57.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.103+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f4b7401ae40 con 0x7f4b7c1363d0 2024-03-29T11:40:57.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.104+0000 7f4b815cd640 10 monclient: get_auth_request con 0x7f4b6006e0b0 auth_method 0 2024-03-29T11:40:57.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.104+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f4b74084b30 con 0x7f4b7c1363d0 2024-03-29T11:40:57.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.104+0000 7f4b71ffb640 10 client.4493.objecter ms_dispatch 0x7f4b7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:57.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.104+0000 7f4b71ffb640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:57.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.104+0000 7f4b71ffb640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:57.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.104+0000 7f4b71ffb640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:40:57.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.104+0000 7f4b71ffb640 10 client.4493.objecter ms_handle_connect 0x7f4b6006e0b0 2024-03-29T11:40:57.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.112+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b74050470 con 0x7f4b7c1363d0 2024-03-29T11:40:57.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.112+0000 7f4b71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:57.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.112+0000 7f4b71ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:57.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.499+0000 7f4b83831640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:57.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.499+0000 7f4b83831640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:57.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.499+0000 7f4b83831640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:40:57.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.499+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b5c005330 con 0x7f4b7c1363d0 2024-03-29T11:40:57.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.501+0000 7f4b71ffb640 1 -- 172.21.15.155:0/3410265071 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f4b740523a0 con 0x7f4b7c1363d0 2024-03-29T11:40:57.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.501+0000 7f4b71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:57.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.501+0000 7f4b71ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:57.504 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:40:57.504 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:57.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.505+0000 7f4b83831640 10 librados: watch_flush enter 2024-03-29T11:40:57.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.505+0000 7f4b83831640 10 librados: watch_flush exit 2024-03-29T11:40:57.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.505+0000 7f4b83831640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:40:57.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.505+0000 7f4b83831640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:40:57.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.506+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 >> v1:172.21.15.155:6816/1760814708 conn(0x7f4b6006e0b0 legacy=0x7f4b60070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.506+0000 7f4b83831640 10 monclient: shutdown 2024-03-29T11:40:57.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.506+0000 7f4b83831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:57.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.506+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 >> v1:172.21.15.155:6790/0 conn(0x7f4b7c1363d0 legacy=0x7f4b7c1e5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.506+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 shutdown_connections 2024-03-29T11:40:57.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.506+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 >> 172.21.15.155:0/3410265071 conn(0x7f4b7c06b660 msgr2=0x7f4b7c13d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:57.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.506+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 shutdown_connections 2024-03-29T11:40:57.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.506+0000 7f4b83831640 1 -- 172.21.15.155:0/3410265071 wait complete. 2024-03-29T11:40:57.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:40:57.506+0000 7f4b83831640 1 librados: shutdown 2024-03-29T11:40:57.524 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:04.331 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:04.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.479+0000 7ff0eb416640 10 monclient: get_monmap_and_config 2024-03-29T11:41:04.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.480+0000 7ff0eb416640 10 monclient: build_initial_monmap 2024-03-29T11:41:04.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.480+0000 7ff0eb416640 10 monclient: monmap: 2024-03-29T11:41:04.484 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:04.484 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:04.484 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:41:04.481452+0000 2024-03-29T11:41:04.484 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:41:04.481452+0000 2024-03-29T11:41:04.484 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:04.484 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:04.484 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:04.484 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:41:04.484 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:41:04.484 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:04.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.481+0000 7ff0eb416640 1 Processor -- start 2024-03-29T11:41:04.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.482+0000 7ff0eb416640 1 -- start start 2024-03-29T11:41:04.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.482+0000 7ff0eb416640 10 monclient: init 2024-03-29T11:41:04.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.483+0000 7ff0eb416640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:04.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.483+0000 7ff0eb416640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:04.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.483+0000 7ff0eb416640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e40a9ac0 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:04.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.483+0000 7ff0eb416640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e413ee90 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:04.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.483+0000 7ff0eb416640 10 monclient(hunting): picked mon.noname-b con 0x7ff0e40af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:04.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.483+0000 7ff0eb416640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e40afd10 con 0x7ff0e413ee90 2024-03-29T11:41:04.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.483+0000 7ff0eb416640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e40af6f0 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.483+0000 7ff0eb416640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e40aac80 con 0x7ff0e40af900 2024-03-29T11:41:04.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.483+0000 7ff0eb416640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:04.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.483+0000 7ff0eb416640 10 monclient(hunting): authenticate will time out at 1246.610326s 2024-03-29T11:41:04.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.487+0000 7ff0e89b1640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff0e413ee90 0x7ff0e413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:43258/0 (socket says 172.21.15.155:43258) 2024-03-29T11:41:04.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.487+0000 7ff0e89b1640 1 -- 172.21.15.155:0/1561603300 learned_addr learned my addr 172.21.15.155:0/1561603300 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:04.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.487+0000 7ff0dbfff640 1 -- 172.21.15.155:0/1561603300 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862208602 0 0) 0x7ff0e40afd10 con 0x7ff0e413ee90 2024-03-29T11:41:04.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.487+0000 7ff0dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:04.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.487+0000 7ff0dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:04.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.487+0000 7ff0dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:04.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.487+0000 7ff0dbfff640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:41:04.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.487+0000 7ff0dbfff640 1 -- 172.21.15.155:0/1561603300 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d0003520 con 0x7ff0e413ee90 2024-03-29T11:41:04.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.488+0000 7ff0dbfff640 1 -- 172.21.15.155:0/1561603300 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059579797 0 0) 0x7ff0d0003520 con 0x7ff0e413ee90 2024-03-29T11:41:04.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.488+0000 7ff0dbfff640 1 -- 172.21.15.155:0/1561603300 >> v1:172.21.15.155:6790/0 conn(0x7ff0e40af900 legacy=0x7ff0e4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:04.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.488+0000 7ff0dbfff640 1 -- 172.21.15.155:0/1561603300 >> v1:172.21.15.183:6789/0 conn(0x7ff0e40a9ac0 legacy=0x7ff0e40adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:04.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:04.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 1 monclient: found mon.noname-a 2024-03-29T11:41:04.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:04.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:41:04.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 1 -- 172.21.15.155:0/1561603300 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e40aaf60 con 0x7ff0e413ee90 2024-03-29T11:41:04.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 10 monclient: _finish_auth 0 2024-03-29T11:41:04.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:04.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:04.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 1 -- 172.21.15.155:0/1561603300 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff0e0001690 con 0x7ff0e413ee90 2024-03-29T11:41:04.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:04.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0dbfff640 10 monclient: dump: 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0eb416640 5 monclient: authenticate success, global_id 4452 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.489+0000 7ff0eb416640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:04.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.490+0000 7ff0dbfff640 1 -- 172.21.15.155:0/1561603300 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0e0001920 con 0x7ff0e413ee90 2024-03-29T11:41:04.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.490+0000 7ff0dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:04.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.490+0000 7ff0eb416640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:04.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.490+0000 7ff0eb416640 4 set_mon_vals no callback set 2024-03-29T11:41:04.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.491+0000 7ff0eb416640 10 monclient: shutdown 2024-03-29T11:41:04.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.491+0000 7ff0eb416640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:04.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.491+0000 7ff0eb416640 1 -- 172.21.15.155:0/1561603300 >> v1:172.21.15.155:6789/0 conn(0x7ff0e413ee90 legacy=0x7ff0e413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:04.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.493+0000 7ff0eb416640 1 -- 172.21.15.155:0/1561603300 shutdown_connections 2024-03-29T11:41:04.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.493+0000 7ff0eb416640 1 -- 172.21.15.155:0/1561603300 >> 172.21.15.155:0/1561603300 conn(0x7ff0e40b35e0 msgr2=0x7ff0e40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:04.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.493+0000 7ff0eb416640 1 -- 172.21.15.155:0/1561603300 shutdown_connections 2024-03-29T11:41:04.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.493+0000 7ff0eb416640 1 -- 172.21.15.155:0/1561603300 wait complete. 2024-03-29T11:41:04.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.494+0000 7ff0eb416640 10 monclient: build_initial_monmap 2024-03-29T11:41:04.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.494+0000 7ff0eb416640 10 monclient: monmap: 2024-03-29T11:41:04.497 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:04.497 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:04.498 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:41:04.498 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:41:04.498 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:04.498 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:04.498 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:04.498 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:41:04.498 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:41:04.498 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.494+0000 7ff0eb416640 1 librados: starting msgr at 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.495+0000 7ff0eb416640 1 librados: starting objecter 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.495+0000 7ff0eb416640 1 Processor -- start 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.495+0000 7ff0eb416640 1 -- start start 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.495+0000 7ff0eb416640 1 librados: setting wanted keys 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.495+0000 7ff0eb416640 1 librados: calling monclient init 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.495+0000 7ff0eb416640 10 monclient: init 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.496+0000 7ff0eb416640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.496+0000 7ff0eb416640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.496+0000 7ff0eb416640 10 monclient(hunting): picked mon.noname-b con 0x7ff0e40a9ac0 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.496+0000 7ff0eb416640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e40af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.496+0000 7ff0eb416640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e413ee90 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:04.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.496+0000 7ff0eb416640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e413f930 con 0x7ff0e413ee90 2024-03-29T11:41:04.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.496+0000 7ff0eb416640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e41dfa20 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.496+0000 7ff0eb416640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e4140460 con 0x7ff0e40af900 2024-03-29T11:41:04.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.496+0000 7ff0eb416640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:04.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.496+0000 7ff0eb416640 10 monclient(hunting): authenticate will time out at 1246.623750s 2024-03-29T11:41:04.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.501+0000 7ff0e99b3640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff0e413ee90 0x7ff0e41df310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48064/0 (socket says 172.21.15.155:48064) 2024-03-29T11:41:04.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.501+0000 7ff0e89b1640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ff0e40af900 0x7ff0e41dec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:38215/0 (socket says 172.21.15.155:38215) 2024-03-29T11:41:04.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.501+0000 7ff0e89b1640 1 -- 172.21.15.155:0/747517322 learned_addr learned my addr 172.21.15.155:0/747517322 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:04.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.501+0000 7ff0d9ffb640 10 client.?.objecter ms_handle_connect 0x7ff0e40af900 2024-03-29T11:41:04.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.501+0000 7ff0d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:04.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.501+0000 7ff0d9ffb640 10 client.?.objecter ms_handle_connect 0x7ff0e40a9ac0 2024-03-29T11:41:04.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.501+0000 7ff0d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:04.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577596305 0 0) 0x7ff0e41dfa20 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c8004eb0 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 client.?.objecter ms_handle_connect 0x7ff0e413ee90 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786781384 0 0) 0x7ff0e4140460 con 0x7ff0e40af900 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e41dfa20 con 0x7ff0e40af900 2024-03-29T11:41:04.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843198110 0 0) 0x7ff0e413f930 con 0x7ff0e413ee90 2024-03-29T11:41:04.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:04.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:04.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:04.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:41:04.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e4140460 con 0x7ff0e413ee90 2024-03-29T11:41:04.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3311051122 0 0) 0x7ff0c8004eb0 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0e413f930 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.502+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff0d40017d0 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:04.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:04.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 10 monclient(hunting): dump: 2024-03-29T11:41:04.512 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:04.512 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:04.512 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:04.512 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:04.512 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:04.512 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:04.513 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:04.513 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:04.513 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:04.513 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:04.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:04.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1772859240 0 0) 0x7ff0e4140460 con 0x7ff0e413ee90 2024-03-29T11:41:04.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0c8004eb0 con 0x7ff0e413ee90 2024-03-29T11:41:04.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff0dc001ae0 con 0x7ff0e413ee90 2024-03-29T11:41:04.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:04.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:04.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 10 monclient(hunting): dump: 2024-03-29T11:41:04.514 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:04.514 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:04.514 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:04.514 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:04.514 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:04.515 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:04.515 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:04.515 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:04.515 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:04.515 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:04.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 242048669 0 0) 0x7ff0e413f930 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 >> v1:172.21.15.155:6790/0 conn(0x7ff0e40af900 legacy=0x7ff0e41dec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 >> v1:172.21.15.155:6789/0 conn(0x7ff0e413ee90 legacy=0x7ff0e41df310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.503+0000 7ff0d9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0d9ffb640 1 monclient: found mon.b 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0d9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e4140740 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0d9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0d9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0eb416640 5 monclient: authenticate success, global_id 4438 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0eb416640 10 monclient: _renew_subs 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0eb416640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0eb416640 1 -- 172.21.15.155:0/747517322 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e413f5c0 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0eb416640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0eb416640 10 monclient: _renew_subs 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0eb416640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0eb416640 1 -- 172.21.15.155:0/747517322 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e4142460 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.504+0000 7ff0eb416640 1 librados: init done 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.506+0000 7ff0cf7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.506+0000 7ff0cf7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.506+0000 7ff0cf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.506+0000 7ff0cf7fe640 1 -- 172.21.15.155:0/747517322 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e4075f10 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.506+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0d4001ce0 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.506+0000 7ff0d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.506+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff0d4003790 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.506+0000 7ff0d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.506+0000 7ff0d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.506+0000 7ff0d9ffb640 10 monclient: dump: 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:04.518 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:04.518 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:04.518 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:04.518 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:04.518 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:04.518 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:04.518 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:04.518 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:04.518 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:04.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.506+0000 7ff0daffd640 4 set_mon_vals no callback set 2024-03-29T11:41:04.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.508+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7ff0d4019760 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.512+0000 7ff0e89b1640 10 monclient: get_auth_request con 0x7ff0c806df30 auth_method 0 2024-03-29T11:41:04.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.513+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7ff0d40836b0 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.513+0000 7ff0d9ffb640 10 client.4438.objecter ms_dispatch 0x7ff0e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:04.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.513+0000 7ff0d9ffb640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:04.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.513+0000 7ff0d9ffb640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:04.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.513+0000 7ff0d9ffb640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-29T11:41:04.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.513+0000 7ff0d9ffb640 10 client.4438.objecter ms_handle_connect 0x7ff0c806df30 2024-03-29T11:41:04.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.514+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0d404edc0 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.514+0000 7ff0d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:04.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.514+0000 7ff0d9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:04.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.867+0000 7ff0cf7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:04.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.867+0000 7ff0cf7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:04.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.867+0000 7ff0cf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:04.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.867+0000 7ff0cf7fe640 1 -- 172.21.15.155:0/747517322 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0e40abbf0 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.868+0000 7ff0d9ffb640 1 -- 172.21.15.155:0/747517322 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7ff0d4050ec0 con 0x7ff0e40a9ac0 2024-03-29T11:41:04.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.868+0000 7ff0d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:04.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.868+0000 7ff0d9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:04.871 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:41:04.872 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:04.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.872+0000 7ff0cf7fe640 10 librados: watch_flush enter 2024-03-29T11:41:04.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.872+0000 7ff0cf7fe640 10 librados: watch_flush exit 2024-03-29T11:41:04.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.872+0000 7ff0cf7fe640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-29T11:41:04.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.872+0000 7ff0cf7fe640 10 client.4438.objecter successfully canceled tick 2024-03-29T11:41:04.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.872+0000 7ff0cf7fe640 1 -- 172.21.15.155:0/747517322 >> v1:172.21.15.155:6816/1760814708 conn(0x7ff0c806df30 legacy=0x7ff0c80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:04.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.872+0000 7ff0cf7fe640 10 monclient: shutdown 2024-03-29T11:41:04.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.872+0000 7ff0cf7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:04.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.872+0000 7ff0cf7fe640 1 -- 172.21.15.155:0/747517322 >> v1:172.21.15.183:6789/0 conn(0x7ff0e40a9ac0 legacy=0x7ff0e4140e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:04.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.875+0000 7ff0cf7fe640 1 -- 172.21.15.155:0/747517322 shutdown_connections 2024-03-29T11:41:04.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.875+0000 7ff0cf7fe640 1 -- 172.21.15.155:0/747517322 >> 172.21.15.155:0/747517322 conn(0x7ff0e40b35e0 msgr2=0x7ff0e40b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:04.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.875+0000 7ff0cf7fe640 1 -- 172.21.15.155:0/747517322 shutdown_connections 2024-03-29T11:41:04.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.875+0000 7ff0cf7fe640 1 -- 172.21.15.155:0/747517322 wait complete. 2024-03-29T11:41:04.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:04.875+0000 7ff0cf7fe640 1 librados: shutdown 2024-03-29T11:41:04.896 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:11.703 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:11.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.841+0000 7fe52e96c640 10 monclient: get_monmap_and_config 2024-03-29T11:41:11.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.841+0000 7fe52e96c640 10 monclient: build_initial_monmap 2024-03-29T11:41:11.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.841+0000 7fe52e96c640 10 monclient: monmap: 2024-03-29T11:41:11.844 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:11.844 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:11.844 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:41:11.842743+0000 2024-03-29T11:41:11.844 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:41:11.842743+0000 2024-03-29T11:41:11.844 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:11.844 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:11.844 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:11.845 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:41:11.845 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:41:11.845 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:11.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.843+0000 7fe52e96c640 1 Processor -- start 2024-03-29T11:41:11.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.843+0000 7fe52e96c640 1 -- start start 2024-03-29T11:41:11.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.843+0000 7fe52e96c640 10 monclient: init 2024-03-29T11:41:11.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.844+0000 7fe52e96c640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:11.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.844+0000 7fe52e96c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:11.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.844+0000 7fe52e96c640 10 monclient(hunting): picked mon.noname-c con 0x7fe5280b0860 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:11.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.844+0000 7fe52e96c640 10 monclient(hunting): picked mon.noname-a con 0x7fe52813ef30 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe52e96c640 10 monclient(hunting): picked mon.noname-b con 0x7fe5280af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe5277fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fe52813ef30 0x7fe5280af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34342/0 (socket says 172.21.15.155:34342) 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe52cf09640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fe5280af8e0 0x7fe528149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:58828/0 (socket says 172.21.15.155:58828) 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe52e96c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5280b0c40 con 0x7fe52813ef30 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe52cf09640 1 -- 172.21.15.155:0/3266899897 learned_addr learned my addr 172.21.15.155:0/3266899897 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe52e96c640 1 -- 172.21.15.155:0/3266899897 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5280aed10 con 0x7fe5280b0860 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe52e96c640 1 -- 172.21.15.155:0/3266899897 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5280afd50 con 0x7fe5280af8e0 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe52e96c640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe52e96c640 10 monclient(hunting): authenticate will time out at 1253.972319s 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 1 -- 172.21.15.155:0/3266899897 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676766950 0 0) 0x7fe5280b0c40 con 0x7fe52813ef30 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:41:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 1 -- 172.21.15.155:0/3266899897 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe510004630 con 0x7fe52813ef30 2024-03-29T11:41:11.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 1 -- 172.21.15.155:0/3266899897 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583489419 0 0) 0x7fe5280afd50 con 0x7fe5280af8e0 2024-03-29T11:41:11.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:11.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:11.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 1 -- 172.21.15.155:0/3266899897 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5280b0c40 con 0x7fe5280af8e0 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 1 -- 172.21.15.155:0/3266899897 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1031043181 0 0) 0x7fe510004630 con 0x7fe52813ef30 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 1 -- 172.21.15.155:0/3266899897 >> v1:172.21.15.155:6790/0 conn(0x7fe5280af8e0 legacy=0x7fe528149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.845+0000 7fe526ffd640 1 -- 172.21.15.155:0/3266899897 >> v1:172.21.15.183:6789/0 conn(0x7fe5280b0860 legacy=0x7fe52813e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 1 monclient: found mon.noname-a 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 1 -- 172.21.15.155:0/3266899897 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5280ab1b0 con 0x7fe52813ef30 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:11.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 1 -- 172.21.15.155:0/3266899897 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe51c001690 con 0x7fe52813ef30 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 10 monclient: dump: 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe52e96c640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe52e96c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:11.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 1 -- 172.21.15.155:0/3266899897 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe51c0017f0 con 0x7fe52813ef30 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 1 -- 172.21.15.155:0/3266899897 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe51c001c40 con 0x7fe52813ef30 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.846+0000 7fe526ffd640 10 monclient: dump: 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:11.854 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:11.855 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:11.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.847+0000 7fe52e96c640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:11.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.847+0000 7fe52e96c640 4 set_mon_vals no callback set 2024-03-29T11:41:11.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.847+0000 7fe52e96c640 10 monclient: shutdown 2024-03-29T11:41:11.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.847+0000 7fe52e96c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:11.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.847+0000 7fe52e96c640 1 -- 172.21.15.155:0/3266899897 >> v1:172.21.15.155:6789/0 conn(0x7fe52813ef30 legacy=0x7fe5280af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:11.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.848+0000 7fe52e96c640 1 -- 172.21.15.155:0/3266899897 shutdown_connections 2024-03-29T11:41:11.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.848+0000 7fe52e96c640 1 -- 172.21.15.155:0/3266899897 >> 172.21.15.155:0/3266899897 conn(0x7fe52806b660 msgr2=0x7fe52806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:11.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.848+0000 7fe52e96c640 1 -- 172.21.15.155:0/3266899897 shutdown_connections 2024-03-29T11:41:11.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.848+0000 7fe52e96c640 1 -- 172.21.15.155:0/3266899897 wait complete. 2024-03-29T11:41:11.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.849+0000 7fe52e96c640 10 monclient: build_initial_monmap 2024-03-29T11:41:11.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.849+0000 7fe52e96c640 10 monclient: monmap: 2024-03-29T11:41:11.861 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:11.861 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:11.861 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:41:11.861 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.851+0000 7fe52e96c640 1 librados: starting msgr at 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.851+0000 7fe52e96c640 1 librados: starting objecter 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.851+0000 7fe52e96c640 1 Processor -- start 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 1 -- start start 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 1 librados: setting wanted keys 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 1 librados: calling monclient init 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 10 monclient: init 2024-03-29T11:41:11.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 10 monclient(hunting): picked mon.noname-b con 0x7fe5280af8e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 10 monclient(hunting): picked mon.noname-c con 0x7fe5280b0860 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 10 monclient(hunting): picked mon.noname-a con 0x7fe52813ef30 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe528071730 con 0x7fe52813ef30 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52806fe10 con 0x7fe5280af8e0 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52806ffa0 con 0x7fe5280b0860 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.852+0000 7fe52e96c640 10 monclient(hunting): authenticate will time out at 1253.979770s 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.853+0000 7fe52cf09640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fe52813ef30 0x7fe5281dab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34756/0 (socket says 172.21.15.155:34756) 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.853+0000 7fe52cf09640 1 -- 172.21.15.155:0/1445538579 learned_addr learned my addr 172.21.15.155:0/1445538579 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.853+0000 7fe524ff9640 10 client.?.objecter ms_handle_connect 0x7fe52813ef30 2024-03-29T11:41:11.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.853+0000 7fe524ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.853+0000 7fe524ff9640 10 client.?.objecter ms_handle_connect 0x7fe5280b0860 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.853+0000 7fe524ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552867960 0 0) 0x7fe528071730 con 0x7fe52813ef30 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe514004eb0 con 0x7fe52813ef30 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1786873164 0 0) 0x7fe514004eb0 con 0x7fe52813ef30 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe528071730 con 0x7fe52813ef30 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe520001dd0 con 0x7fe52813ef30 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): dump: 2024-03-29T11:41:11.864 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119555453 0 0) 0x7fe52806ffa0 con 0x7fe5280b0860 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.854+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe514004eb0 con 0x7fe5280b0860 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.855+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 771226800 0 0) 0x7fe528071730 con 0x7fe52813ef30 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.855+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 >> v1:172.21.15.155:6790/0 conn(0x7fe5280b0860 legacy=0x7fe528134820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.855+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 >> v1:172.21.15.183:6789/0 conn(0x7fe5280af8e0 legacy=0x7fe52806f700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 1 monclient: found mon.a 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52806eeb0 con 0x7fe52813ef30 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 10 monclient: _finish_auth 0 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe52e96c640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:41:11.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe52e96c640 10 monclient: _renew_subs 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe52e96c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe52e96c640 1 -- 172.21.15.155:0/1445538579 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52806f120 con 0x7fe52813ef30 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe520004b60 con 0x7fe52813ef30 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fe520004e70 con 0x7fe52813ef30 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe524ff9640 10 monclient: dump: 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe525ffb640 4 set_mon_vals no callback set 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe52e96c640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe52e96c640 10 monclient: _renew_subs 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe52e96c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe52e96c640 1 -- 172.21.15.155:0/1445538579 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe528135110 con 0x7fe52813ef30 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.856+0000 7fe52e96c640 1 librados: init done 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.858+0000 7fe52e96c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.858+0000 7fe52e96c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.858+0000 7fe52e96c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.858+0000 7fe52e96c640 1 -- 172.21.15.155:0/1445538579 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe528071550 con 0x7fe52813ef30 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.859+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fe52001ae40 con 0x7fe52813ef30 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.859+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fe520084e70 con 0x7fe52813ef30 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.859+0000 7fe524ff9640 10 client.4461.objecter ms_dispatch 0x7fe528000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.859+0000 7fe524ff9640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.859+0000 7fe524ff9640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.859+0000 7fe524ff9640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.860+0000 7fe527fff640 10 monclient: get_auth_request con 0x7fe51406ddc0 auth_method 0 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.860+0000 7fe524ff9640 10 client.4461.objecter ms_handle_connect 0x7fe51406ddc0 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.866+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe520050750 con 0x7fe52813ef30 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.866+0000 7fe524ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:11.866+0000 7fe524ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:12.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.183+0000 7fe52e96c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:12.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.183+0000 7fe52e96c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:12.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.183+0000 7fe52e96c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:12.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.183+0000 7fe52e96c640 1 -- 172.21.15.155:0/1445538579 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe528075f10 con 0x7fe52813ef30 2024-03-29T11:41:12.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.184+0000 7fe524ff9640 1 -- 172.21.15.155:0/1445538579 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fe520052680 con 0x7fe52813ef30 2024-03-29T11:41:12.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.184+0000 7fe524ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:12.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.184+0000 7fe524ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:12.188 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:41:12.188 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.187+0000 7fe52e96c640 10 librados: watch_flush enter 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.187+0000 7fe52e96c640 10 librados: watch_flush exit 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.187+0000 7fe52e96c640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.187+0000 7fe52e96c640 10 client.4461.objecter successfully canceled tick 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.187+0000 7fe52e96c640 1 -- 172.21.15.155:0/1445538579 >> v1:172.21.15.155:6816/1760814708 conn(0x7fe51406ddc0 legacy=0x7fe514070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.187+0000 7fe52e96c640 10 monclient: shutdown 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.187+0000 7fe52e96c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.187+0000 7fe52e96c640 1 -- 172.21.15.155:0/1445538579 >> v1:172.21.15.155:6789/0 conn(0x7fe52813ef30 legacy=0x7fe5281dab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.188+0000 7fe52e96c640 1 -- 172.21.15.155:0/1445538579 shutdown_connections 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.188+0000 7fe52e96c640 1 -- 172.21.15.155:0/1445538579 >> 172.21.15.155:0/1445538579 conn(0x7fe52806b660 msgr2=0x7fe5280ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.188+0000 7fe52e96c640 1 -- 172.21.15.155:0/1445538579 shutdown_connections 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.188+0000 7fe52e96c640 1 -- 172.21.15.155:0/1445538579 wait complete. 2024-03-29T11:41:12.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:12.188+0000 7fe52e96c640 1 librados: shutdown 2024-03-29T11:41:12.205 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:19.012 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:19.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.147+0000 7f6c7cb13640 10 monclient: get_monmap_and_config 2024-03-29T11:41:19.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.147+0000 7f6c7cb13640 10 monclient: build_initial_monmap 2024-03-29T11:41:19.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.147+0000 7f6c7cb13640 10 monclient: monmap: 2024-03-29T11:41:19.151 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:19.151 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:19.151 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:41:19.148509+0000 2024-03-29T11:41:19.151 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:41:19.148509+0000 2024-03-29T11:41:19.151 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:19.152 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:19.152 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:19.152 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:41:19.152 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:41:19.152 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:19.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.148+0000 7f6c7cb13640 1 Processor -- start 2024-03-29T11:41:19.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.148+0000 7f6c7cb13640 1 -- start start 2024-03-29T11:41:19.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.148+0000 7f6c7cb13640 10 monclient: init 2024-03-29T11:41:19.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.149+0000 7f6c7cb13640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:19.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.149+0000 7f6c7cb13640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:19.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.149+0000 7f6c7cb13640 10 monclient(hunting): picked mon.noname-a con 0x7f6c780a9ac0 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:19.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.149+0000 7f6c7cb13640 10 monclient(hunting): picked mon.noname-c con 0x7f6c7813ee90 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:19.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.149+0000 7f6c7cb13640 10 monclient(hunting): picked mon.noname-b con 0x7f6c780af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:19.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.149+0000 7f6c76d9d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f6c780af900 0x7f6c78149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:37154/0 (socket says 172.21.15.155:37154) 2024-03-29T11:41:19.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.149+0000 7f6c76d9d640 1 -- 172.21.15.155:0/4208230893 learned_addr learned my addr 172.21.15.155:0/4208230893 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:19.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.149+0000 7f6c7cb13640 1 -- 172.21.15.155:0/4208230893 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c780afd10 con 0x7f6c780a9ac0 2024-03-29T11:41:19.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.149+0000 7f6c7cb13640 1 -- 172.21.15.155:0/4208230893 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c780af6f0 con 0x7f6c7813ee90 2024-03-29T11:41:19.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.149+0000 7f6c7cb13640 1 -- 172.21.15.155:0/4208230893 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c780aac80 con 0x7f6c780af900 2024-03-29T11:41:19.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.150+0000 7f6c7cb13640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:19.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.150+0000 7f6c7cb13640 10 monclient(hunting): authenticate will time out at 1261.277000s 2024-03-29T11:41:19.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.150+0000 7f6c7559a640 1 -- 172.21.15.155:0/4208230893 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157534030 0 0) 0x7f6c780aac80 con 0x7f6c780af900 2024-03-29T11:41:19.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.150+0000 7f6c7559a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:19.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.150+0000 7f6c7559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:19.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.150+0000 7f6c7559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:19.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.150+0000 7f6c7559a640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:41:19.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.150+0000 7f6c7559a640 1 -- 172.21.15.155:0/4208230893 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c60003440 con 0x7f6c780af900 2024-03-29T11:41:19.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.150+0000 7f6c7559a640 1 -- 172.21.15.155:0/4208230893 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3074514630 0 0) 0x7f6c60003440 con 0x7f6c780af900 2024-03-29T11:41:19.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.150+0000 7f6c7559a640 1 -- 172.21.15.155:0/4208230893 >> v1:172.21.15.183:6789/0 conn(0x7f6c7813ee90 legacy=0x7f6c7813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:19.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.150+0000 7f6c7559a640 1 -- 172.21.15.155:0/4208230893 >> v1:172.21.15.155:6789/0 conn(0x7f6c780a9ac0 legacy=0x7f6c780adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 10 monclient: _finish_hunting 0 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 1 monclient: found mon.noname-b 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 1 -- 172.21.15.155:0/4208230893 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c780aaf60 con 0x7f6c780af900 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 10 monclient: _finish_auth 0 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 10 monclient: _check_auth_tickets 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 1 -- 172.21.15.155:0/4208230893 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6c68004cb0 con 0x7f6c780af900 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 10 monclient: dump: 2024-03-29T11:41:19.158 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7cb13640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7cb13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 1 -- 172.21.15.155:0/4208230893 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c68004e80 con 0x7f6c780af900 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 1 -- 172.21.15.155:0/4208230893 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6c68001430 con 0x7f6c780af900 2024-03-29T11:41:19.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.153+0000 7f6c7559a640 10 monclient: dump: 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.154+0000 7f6c7cb13640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:19.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.154+0000 7f6c7cb13640 4 set_mon_vals no callback set 2024-03-29T11:41:19.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.156+0000 7f6c7cb13640 10 monclient: shutdown 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.156+0000 7f6c7cb13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.156+0000 7f6c7cb13640 1 -- 172.21.15.155:0/4208230893 >> v1:172.21.15.155:6790/0 conn(0x7f6c780af900 legacy=0x7f6c78149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.156+0000 7f6c7cb13640 1 -- 172.21.15.155:0/4208230893 shutdown_connections 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.156+0000 7f6c7cb13640 1 -- 172.21.15.155:0/4208230893 >> 172.21.15.155:0/4208230893 conn(0x7f6c780b35e0 msgr2=0x7f6c780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.156+0000 7f6c7cb13640 1 -- 172.21.15.155:0/4208230893 shutdown_connections 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.157+0000 7f6c7cb13640 1 -- 172.21.15.155:0/4208230893 wait complete. 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.158+0000 7f6c7cb13640 10 monclient: build_initial_monmap 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.158+0000 7f6c7cb13640 10 monclient: monmap: 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:19.163 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.158+0000 7f6c7cb13640 1 librados: starting msgr at 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.158+0000 7f6c7cb13640 1 librados: starting objecter 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.158+0000 7f6c7cb13640 1 Processor -- start 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.159+0000 7f6c7cb13640 1 -- start start 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.159+0000 7f6c7cb13640 1 librados: setting wanted keys 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.159+0000 7f6c7cb13640 1 librados: calling monclient init 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.159+0000 7f6c7cb13640 10 monclient: init 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7cb13640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7cb13640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7cb13640 10 monclient(hunting): picked mon.noname-b con 0x7f6c780a9ac0 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:19.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7cb13640 10 monclient(hunting): picked mon.noname-c con 0x7f6c780af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:19.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7659c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f6c780a9ac0 0x7f6c7813f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:57950/0 (socket says 172.21.15.155:57950) 2024-03-29T11:41:19.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7659c640 1 -- 172.21.15.155:0/3145416838 learned_addr learned my addr 172.21.15.155:0/3145416838 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:19.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7cb13640 10 monclient(hunting): picked mon.noname-a con 0x7f6c7813ee90 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:19.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c781d6770 con 0x7f6c7813ee90 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c78071aa0 con 0x7f6c780a9ac0 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c78071c30 con 0x7f6c780af900 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c66ffd640 10 client.?.objecter ms_handle_connect 0x7f6c780a9ac0 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7cb13640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.160+0000 7f6c7cb13640 10 monclient(hunting): authenticate will time out at 1261.287850s 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.161+0000 7f6c66ffd640 10 client.?.objecter ms_handle_connect 0x7f6c780af900 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.161+0000 7f6c66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825608960 0 0) 0x7f6c78071c30 con 0x7f6c780af900 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c58004eb0 con 0x7f6c780af900 2024-03-29T11:41:19.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1560621949 0 0) 0x7f6c58004eb0 con 0x7f6c780af900 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c78071c30 con 0x7f6c780af900 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6c6c001870 con 0x7f6c780af900 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:19.173 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.162+0000 7f6c66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 10 client.?.objecter ms_handle_connect 0x7f6c7813ee90 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675409495 0 0) 0x7f6c78071aa0 con 0x7f6c780a9ac0 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c58004eb0 con 0x7f6c780a9ac0 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190596235 0 0) 0x7f6c781d6770 con 0x7f6c7813ee90 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:19.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c78071aa0 con 0x7f6c7813ee90 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 375961901 0 0) 0x7f6c78071aa0 con 0x7f6c7813ee90 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c781d6770 con 0x7f6c7813ee90 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.163+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6c68004b70 con 0x7f6c7813ee90 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:19.175 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3618781415 0 0) 0x7f6c58004eb0 con 0x7f6c780a9ac0 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c78071aa0 con 0x7f6c780a9ac0 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6c70001a10 con 0x7f6c780a9ac0 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:19.176 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3896454734 0 0) 0x7f6c781d6770 con 0x7f6c7813ee90 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 >> v1:172.21.15.155:6790/0 conn(0x7f6c780af900 legacy=0x7f6c78138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 >> v1:172.21.15.183:6789/0 conn(0x7f6c780a9ac0 legacy=0x7f6c7813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 1 monclient: found mon.a 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c78071dc0 con 0x7f6c7813ee90 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:19.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c7cb13640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c7cb13640 10 monclient: _renew_subs 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c7cb13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c781d6100 con 0x7f6c7813ee90 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c7cb13640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c7cb13640 10 monclient: _renew_subs 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c7cb13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c780b26c0 con 0x7f6c7813ee90 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.164+0000 7f6c7cb13640 1 librados: init done 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.165+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c68004cd0 con 0x7f6c7813ee90 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.165+0000 7f6c66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.165+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6c68001b40 con 0x7f6c7813ee90 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.165+0000 7f6c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.165+0000 7f6c66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.165+0000 7f6c66ffd640 10 monclient: dump: 2024-03-29T11:41:19.178 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.165+0000 7f6c74d99640 4 set_mon_vals no callback set 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.166+0000 7f6c7cb13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.166+0000 7f6c7cb13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.166+0000 7f6c7cb13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.166+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c005080 con 0x7f6c7813ee90 2024-03-29T11:41:19.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.166+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f6c6801ae60 con 0x7f6c7813ee90 2024-03-29T11:41:19.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.167+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f6c68084a50 con 0x7f6c7813ee90 2024-03-29T11:41:19.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.167+0000 7f6c66ffd640 10 client.4464.objecter ms_dispatch 0x7f6c78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:19.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.167+0000 7f6c66ffd640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:19.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.167+0000 7f6c66ffd640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:19.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.167+0000 7f6c66ffd640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-29T11:41:19.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.167+0000 7f6c7659c640 10 monclient: get_auth_request con 0x7f6c5806e0f0 auth_method 0 2024-03-29T11:41:19.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.167+0000 7f6c66ffd640 10 client.4464.objecter ms_handle_connect 0x7f6c5806e0f0 2024-03-29T11:41:19.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.173+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c68050330 con 0x7f6c7813ee90 2024-03-29T11:41:19.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.173+0000 7f6c66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:19.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.173+0000 7f6c66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:19.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.476+0000 7f6c7cb13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:19.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.476+0000 7f6c7cb13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:19.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.476+0000 7f6c7cb13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:19.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.476+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c5c005330 con 0x7f6c7813ee90 2024-03-29T11:41:19.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.478+0000 7f6c66ffd640 1 -- 172.21.15.155:0/3145416838 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f6c68052260 con 0x7f6c7813ee90 2024-03-29T11:41:19.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.478+0000 7f6c66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:19.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.478+0000 7f6c66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:19.481 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:41:19.481 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:19.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 10 librados: watch_flush enter 2024-03-29T11:41:19.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 10 librados: watch_flush exit 2024-03-29T11:41:19.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-29T11:41:19.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 10 client.4464.objecter successfully canceled tick 2024-03-29T11:41:19.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 >> v1:172.21.15.155:6816/1760814708 conn(0x7f6c5806e0f0 legacy=0x7f6c580705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 10 monclient: shutdown 2024-03-29T11:41:19.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:19.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 >> v1:172.21.15.155:6789/0 conn(0x7f6c7813ee90 legacy=0x7f6c781de4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 shutdown_connections 2024-03-29T11:41:19.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 >> 172.21.15.155:0/3145416838 conn(0x7f6c780b35e0 msgr2=0x7f6c780adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:19.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 shutdown_connections 2024-03-29T11:41:19.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.481+0000 7f6c7cb13640 1 -- 172.21.15.155:0/3145416838 wait complete. 2024-03-29T11:41:19.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:19.482+0000 7f6c7cb13640 1 librados: shutdown 2024-03-29T11:41:19.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:26.307 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:26.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.462+0000 7fefb9095640 10 monclient: get_monmap_and_config 2024-03-29T11:41:26.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.462+0000 7fefb9095640 10 monclient: build_initial_monmap 2024-03-29T11:41:26.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.462+0000 7fefb9095640 10 monclient: monmap: 2024-03-29T11:41:26.465 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:26.465 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:26.465 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:41:26.463911+0000 2024-03-29T11:41:26.465 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:41:26.463911+0000 2024-03-29T11:41:26.465 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:26.466 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:26.466 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:26.466 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:41:26.466 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:41:26.466 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:26.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.465+0000 7fefb9095640 1 Processor -- start 2024-03-29T11:41:26.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.465+0000 7fefb9095640 1 -- start start 2024-03-29T11:41:26.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.465+0000 7fefb9095640 10 monclient: init 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.466+0000 7fefb9095640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.466+0000 7fefb9095640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.466+0000 7fefb9095640 10 monclient(hunting): picked mon.noname-a con 0x7fefb40b4330 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.466+0000 7fefb9095640 10 monclient(hunting): picked mon.noname-c con 0x7fefb41470b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.466+0000 7fefb9095640 10 monclient(hunting): picked mon.noname-b con 0x7fefb41439b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.466+0000 7fefb9095640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb406f2a0 con 0x7fefb40b4330 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.466+0000 7fefb9095640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb4146b90 con 0x7fefb41470b0 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.466+0000 7fefb9095640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb40b2ca0 con 0x7fefb41439b0 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.466+0000 7fefb9095640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.466+0000 7fefb9095640 10 monclient(hunting): authenticate will time out at 1268.593355s 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.469+0000 7fefb259c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fefb41470b0 0x7fefb40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:40949/0 (socket says 172.21.15.155:40949) 2024-03-29T11:41:26.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.469+0000 7fefb259c640 1 -- 172.21.15.155:0/3487524878 learned_addr learned my addr 172.21.15.155:0/3487524878 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:26.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.469+0000 7fefb1d9b640 1 -- 172.21.15.155:0/3487524878 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207931175 0 0) 0x7fefb4146b90 con 0x7fefb41470b0 2024-03-29T11:41:26.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:26.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 1 -- 172.21.15.155:0/3487524878 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c003520 con 0x7fefb41470b0 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 1 -- 172.21.15.155:0/3487524878 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870903294 0 0) 0x7fefb40b2ca0 con 0x7fefb41439b0 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 1 -- 172.21.15.155:0/3487524878 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb4146b90 con 0x7fefb41439b0 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 1 -- 172.21.15.155:0/3487524878 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2242824247 0 0) 0x7fef9c003520 con 0x7fefb41470b0 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 1 -- 172.21.15.155:0/3487524878 >> v1:172.21.15.155:6790/0 conn(0x7fefb41439b0 legacy=0x7fefb4151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:26.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.470+0000 7fefb1d9b640 1 -- 172.21.15.155:0/3487524878 >> v1:172.21.15.155:6789/0 conn(0x7fefb40b4330 legacy=0x7fefb40b4710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:26.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:41:26.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 1 monclient: found mon.noname-c 2024-03-29T11:41:26.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:26.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:41:26.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 1 -- 172.21.15.155:0/3487524878 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb406e0c0 con 0x7fefb41470b0 2024-03-29T11:41:26.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:41:26.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:41:26.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:26.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 1 -- 172.21.15.155:0/3487524878 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fefa4001690 con 0x7fefb41470b0 2024-03-29T11:41:26.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:26.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:26.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 10 monclient: dump: 2024-03-29T11:41:26.477 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:26.477 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:26.477 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:26.477 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:26.477 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:26.477 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:26.477 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:26.478 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:26.478 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:26.478 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:26.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb9095640 5 monclient: authenticate success, global_id 4444 2024-03-29T11:41:26.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb9095640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:26.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 1 -- 172.21.15.155:0/3487524878 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefa40017f0 con 0x7fefb41470b0 2024-03-29T11:41:26.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:26.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 1 -- 172.21.15.155:0/3487524878 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fefa4001c00 con 0x7fefb41470b0 2024-03-29T11:41:26.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.472+0000 7fefb1d9b640 10 monclient: dump: 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.473+0000 7fefb9095640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:26.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.473+0000 7fefb9095640 4 set_mon_vals no callback set 2024-03-29T11:41:26.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.474+0000 7fefb9095640 10 monclient: shutdown 2024-03-29T11:41:26.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.474+0000 7fefb9095640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:26.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.474+0000 7fefb9095640 1 -- 172.21.15.155:0/3487524878 >> v1:172.21.15.183:6789/0 conn(0x7fefb41470b0 legacy=0x7fefb40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:26.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.474+0000 7fefb9095640 1 -- 172.21.15.155:0/3487524878 shutdown_connections 2024-03-29T11:41:26.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.474+0000 7fefb9095640 1 -- 172.21.15.155:0/3487524878 >> 172.21.15.155:0/3487524878 conn(0x7fefb4066a00 msgr2=0x7fefb4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:26.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.474+0000 7fefb9095640 1 -- 172.21.15.155:0/3487524878 shutdown_connections 2024-03-29T11:41:26.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.474+0000 7fefb9095640 1 -- 172.21.15.155:0/3487524878 wait complete. 2024-03-29T11:41:26.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.476+0000 7fefb9095640 10 monclient: build_initial_monmap 2024-03-29T11:41:26.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.476+0000 7fefb9095640 10 monclient: monmap: 2024-03-29T11:41:26.481 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:26.481 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.476+0000 7fefb9095640 1 librados: starting msgr at 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.476+0000 7fefb9095640 1 librados: starting objecter 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.477+0000 7fefb9095640 1 Processor -- start 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.477+0000 7fefb9095640 1 -- start start 2024-03-29T11:41:26.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.477+0000 7fefb9095640 1 librados: setting wanted keys 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.477+0000 7fefb9095640 1 librados: calling monclient init 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.477+0000 7fefb9095640 10 monclient: init 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.477+0000 7fefb9095640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.477+0000 7fefb9095640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.478+0000 7fefb9095640 10 monclient(hunting): picked mon.noname-b con 0x7fefb40b4330 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.478+0000 7fefb9095640 10 monclient(hunting): picked mon.noname-c con 0x7fefb41439b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.478+0000 7fefb9095640 10 monclient(hunting): picked mon.noname-a con 0x7fefb41470b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.478+0000 7fefb9095640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb4121550 con 0x7fefb41470b0 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.478+0000 7fefb9095640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb41dbbc0 con 0x7fefb40b4330 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.478+0000 7fefb9095640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb41dbd50 con 0x7fefb41439b0 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.478+0000 7fefb9095640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:26.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.478+0000 7fefb9095640 10 monclient(hunting): authenticate will time out at 1268.604939s 2024-03-29T11:41:26.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.478+0000 7fefb259c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fefb41439b0 0x7fefb41db4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:41168/0 (socket says 172.21.15.155:41168) 2024-03-29T11:41:26.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.479+0000 7fefb259c640 1 -- 172.21.15.155:0/79714769 learned_addr learned my addr 172.21.15.155:0/79714769 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.480+0000 7fef9b7fe640 10 client.?.objecter ms_handle_connect 0x7fefb41439b0 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.480+0000 7fef9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402780348 0 0) 0x7fefb41dbd50 con 0x7fefb41439b0 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa0004eb0 con 0x7fefb41439b0 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2996306080 0 0) 0x7fefa0004eb0 con 0x7fefb41439b0 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefb41dbd50 con 0x7fefb41439b0 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fefa4001c40 con 0x7fefb41439b0 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 10 monclient(hunting): dump: 2024-03-29T11:41:26.485 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3635125011 0 0) 0x7fefb41dbd50 con 0x7fefb41439b0 2024-03-29T11:41:26.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.481+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 >> v1:172.21.15.183:6789/0 conn(0x7fefb40b4330 legacy=0x7fefb411f9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:26.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.482+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 >> v1:172.21.15.155:6789/0 conn(0x7fefb41470b0 legacy=0x7fefb41e5f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:26.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.483+0000 7fef9b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:26.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.483+0000 7fef9b7fe640 1 monclient: found mon.c 2024-03-29T11:41:26.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.483+0000 7fef9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:26.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.483+0000 7fef9b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:41:26.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.483+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb41415e0 con 0x7fefb41439b0 2024-03-29T11:41:26.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.483+0000 7fef9b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:26.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.483+0000 7fef9b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:26.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.483+0000 7fef9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.484+0000 7fefb9095640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.484+0000 7fefb9095640 10 monclient: _renew_subs 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.484+0000 7fefb9095640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.484+0000 7fefb9095640 1 -- 172.21.15.155:0/79714769 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb41203d0 con 0x7fefb41439b0 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.484+0000 7fefb9095640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.484+0000 7fefb9095640 10 monclient: _renew_subs 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.484+0000 7fefb9095640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.484+0000 7fefb9095640 1 -- 172.21.15.155:0/79714769 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb41d10f0 con 0x7fefb41439b0 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.484+0000 7fefb9095640 1 librados: init done 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.485+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefa4004a90 con 0x7fefb41439b0 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.485+0000 7fef9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.485+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fefa4004ea0 con 0x7fefb41439b0 2024-03-29T11:41:26.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.485+0000 7fef9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.485+0000 7fef9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.485+0000 7fef9b7fe640 10 monclient: dump: 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.485+0000 7fefb0d99640 4 set_mon_vals no callback set 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.485+0000 7fefb9095640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:26.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.486+0000 7fefb9095640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:26.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.486+0000 7fefb9095640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:41:26.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.486+0000 7fefb9095640 1 -- 172.21.15.155:0/79714769 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef94005080 con 0x7fefb41439b0 2024-03-29T11:41:26.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.486+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fefa4005000 con 0x7fefb41439b0 2024-03-29T11:41:26.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.487+0000 7fefb2d9d640 10 monclient: get_auth_request con 0x7fefa006dc80 auth_method 0 2024-03-29T11:41:26.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.488+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fefa4084c90 con 0x7fefb41439b0 2024-03-29T11:41:26.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.488+0000 7fef9b7fe640 10 client.4514.objecter ms_dispatch 0x7fefb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:26.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.488+0000 7fef9b7fe640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:26.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.488+0000 7fef9b7fe640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:26.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.488+0000 7fef9b7fe640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-29T11:41:26.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.488+0000 7fef9b7fe640 10 client.4514.objecter ms_handle_connect 0x7fefa006dc80 2024-03-29T11:41:26.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.495+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefa40505d0 con 0x7fefb41439b0 2024-03-29T11:41:26.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.495+0000 7fef9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:26.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.495+0000 7fef9b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:26.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.823+0000 7fefb9095640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:26.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.823+0000 7fefb9095640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:26.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.823+0000 7fefb9095640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:41:26.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.823+0000 7fefb9095640 1 -- 172.21.15.155:0/79714769 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef94005330 con 0x7fefb41439b0 2024-03-29T11:41:26.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.824+0000 7fef9b7fe640 1 -- 172.21.15.155:0/79714769 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fefa4084340 con 0x7fefb41439b0 2024-03-29T11:41:26.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.824+0000 7fef9b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:26.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.824+0000 7fef9b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:26.828 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:41:26.828 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.828+0000 7fefb9095640 10 librados: watch_flush enter 2024-03-29T11:41:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 10 librados: watch_flush exit 2024-03-29T11:41:26.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-29T11:41:26.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 10 client.4514.objecter successfully canceled tick 2024-03-29T11:41:26.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 1 -- 172.21.15.155:0/79714769 >> v1:172.21.15.155:6816/1760814708 conn(0x7fefa006dc80 legacy=0x7fefa0070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:26.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 10 monclient: shutdown 2024-03-29T11:41:26.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:26.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 1 -- 172.21.15.155:0/79714769 >> v1:172.21.15.155:6790/0 conn(0x7fefb41439b0 legacy=0x7fefb41db4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:26.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 1 -- 172.21.15.155:0/79714769 shutdown_connections 2024-03-29T11:41:26.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 1 -- 172.21.15.155:0/79714769 >> 172.21.15.155:0/79714769 conn(0x7fefb4066a00 msgr2=0x7fefb40b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:26.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 1 -- 172.21.15.155:0/79714769 shutdown_connections 2024-03-29T11:41:26.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 1 -- 172.21.15.155:0/79714769 wait complete. 2024-03-29T11:41:26.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:26.829+0000 7fefb9095640 1 librados: shutdown 2024-03-29T11:41:26.848 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:33.655 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:33.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.807+0000 7fca251bc640 10 monclient: get_monmap_and_config 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.807+0000 7fca251bc640 10 monclient: build_initial_monmap 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.807+0000 7fca251bc640 10 monclient: monmap: 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:41:33.808219+0000 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:41:33.808219+0000 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:41:33.810 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:33.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.810+0000 7fca251bc640 1 Processor -- start 2024-03-29T11:41:33.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.810+0000 7fca251bc640 1 -- start start 2024-03-29T11:41:33.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.810+0000 7fca251bc640 10 monclient: init 2024-03-29T11:41:33.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.811+0000 7fca251bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:33.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.811+0000 7fca251bc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.811+0000 7fca251bc640 10 monclient(hunting): picked mon.noname-b con 0x7fca200b2800 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.811+0000 7fca251bc640 10 monclient(hunting): picked mon.noname-a con 0x7fca200a9c00 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.811+0000 7fca251bc640 10 monclient(hunting): picked mon.noname-c con 0x7fca200b38c0 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.811+0000 7fca251bc640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca20141eb0 con 0x7fca200a9c00 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.811+0000 7fca251bc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca200b22d0 con 0x7fca200b38c0 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.811+0000 7fca251bc640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca20142d30 con 0x7fca200b2800 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.811+0000 7fca251bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.811+0000 7fca251bc640 10 monclient(hunting): authenticate will time out at 1275.937808s 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.812+0000 7fca1ffff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fca200b2800 0x7fca20142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:34722/0 (socket says 172.21.15.155:34722) 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.812+0000 7fca1ffff640 1 -- 172.21.15.155:0/2788951767 learned_addr learned my addr 172.21.15.155:0/2788951767 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.812+0000 7fca1effd640 1 -- 172.21.15.155:0/2788951767 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593817041 0 0) 0x7fca20142d30 con 0x7fca200b2800 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.812+0000 7fca1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.812+0000 7fca1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.812+0000 7fca1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:33.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.812+0000 7fca1effd640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:41:33.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.813+0000 7fca1effd640 1 -- 172.21.15.155:0/2788951767 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca08003520 con 0x7fca200b2800 2024-03-29T11:41:33.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.815+0000 7fca1effd640 1 -- 172.21.15.155:0/2788951767 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2350374394 0 0) 0x7fca08003520 con 0x7fca200b2800 2024-03-29T11:41:33.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.815+0000 7fca1effd640 1 -- 172.21.15.155:0/2788951767 >> v1:172.21.15.183:6789/0 conn(0x7fca200b38c0 legacy=0x7fca2014d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:33.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.815+0000 7fca1effd640 1 -- 172.21.15.155:0/2788951767 >> v1:172.21.15.155:6789/0 conn(0x7fca200a9c00 legacy=0x7fca2013bb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:33.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.815+0000 7fca1effd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:33.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca1effd640 1 monclient: found mon.noname-b 2024-03-29T11:41:33.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:33.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca1effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca1effd640 1 -- 172.21.15.155:0/2788951767 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca200aaf90 con 0x7fca200b2800 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca1effd640 10 monclient: _finish_auth 0 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca1effd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca251bc640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca251bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca1effd640 1 -- 172.21.15.155:0/2788951767 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fca10001690 con 0x7fca200b2800 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca1effd640 10 monclient: dump: 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:33.819 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:33.820 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:33.820 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:33.820 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:33.820 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:33.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.816+0000 7fca251bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:33.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.818+0000 7fca1effd640 1 -- 172.21.15.155:0/2788951767 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca100017f0 con 0x7fca200b2800 2024-03-29T11:41:33.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.818+0000 7fca1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:33.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.818+0000 7fca1effd640 1 -- 172.21.15.155:0/2788951767 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fca10001c40 con 0x7fca200b2800 2024-03-29T11:41:33.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.818+0000 7fca1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:33.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.818+0000 7fca1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:33.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.818+0000 7fca1effd640 10 monclient: dump: 2024-03-29T11:41:33.821 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:33.821 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:33.821 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:33.821 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:33.821 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:33.821 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:33.821 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:33.821 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:33.821 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:33.821 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:33.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.819+0000 7fca251bc640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:33.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.819+0000 7fca251bc640 4 set_mon_vals no callback set 2024-03-29T11:41:33.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.819+0000 7fca251bc640 10 monclient: shutdown 2024-03-29T11:41:33.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.819+0000 7fca251bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:33.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.819+0000 7fca251bc640 1 -- 172.21.15.155:0/2788951767 >> v1:172.21.15.155:6790/0 conn(0x7fca200b2800 legacy=0x7fca20142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:33.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.819+0000 7fca251bc640 1 -- 172.21.15.155:0/2788951767 shutdown_connections 2024-03-29T11:41:33.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.819+0000 7fca251bc640 1 -- 172.21.15.155:0/2788951767 >> 172.21.15.155:0/2788951767 conn(0x7fca20066a00 msgr2=0x7fca20066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:33.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.820+0000 7fca251bc640 1 -- 172.21.15.155:0/2788951767 shutdown_connections 2024-03-29T11:41:33.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.821+0000 7fca251bc640 1 -- 172.21.15.155:0/2788951767 wait complete. 2024-03-29T11:41:33.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.822+0000 7fca251bc640 10 monclient: build_initial_monmap 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.822+0000 7fca251bc640 10 monclient: monmap: 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.822+0000 7fca251bc640 1 librados: starting msgr at 2024-03-29T11:41:33.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.822+0000 7fca251bc640 1 librados: starting objecter 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.823+0000 7fca251bc640 1 Processor -- start 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.823+0000 7fca251bc640 1 -- start start 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.823+0000 7fca251bc640 1 librados: setting wanted keys 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.823+0000 7fca251bc640 1 librados: calling monclient init 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.823+0000 7fca251bc640 10 monclient: init 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca251bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca251bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca251bc640 10 monclient(hunting): picked mon.noname-b con 0x7fca200a9c00 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca251bc640 10 monclient(hunting): picked mon.noname-a con 0x7fca200b2800 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca251bc640 10 monclient(hunting): picked mon.noname-c con 0x7fca200b38c0 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca251bc640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2006f2f0 con 0x7fca200b2800 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca251bc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2006e350 con 0x7fca200a9c00 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca251bc640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca20145e90 con 0x7fca200b38c0 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca251bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:33.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca251bc640 10 monclient(hunting): authenticate will time out at 1275.950082s 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca249bb640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fca200b38c0 0x7fca201e44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36830/0 (socket says 172.21.15.155:36830) 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca249bb640 1 -- 172.21.15.155:0/4154068785 learned_addr learned my addr 172.21.15.155:0/4154068785 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca1cff9640 10 client.?.objecter ms_handle_connect 0x7fca200b38c0 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.824+0000 7fca1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.825+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303886174 0 0) 0x7fca20145e90 con 0x7fca200b38c0 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.825+0000 7fca1cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.825+0000 7fca1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.825+0000 7fca1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.825+0000 7fca1cff9640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.825+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c004eb0 con 0x7fca200b38c0 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.827+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3591004057 0 0) 0x7fca0c004eb0 con 0x7fca200b38c0 2024-03-29T11:41:33.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.827+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca20145e90 con 0x7fca200b38c0 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.827+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fca18001ad0 con 0x7fca200b38c0 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.827+0000 7fca1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.827+0000 7fca1cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.827+0000 7fca1cff9640 10 monclient(hunting): dump: 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:33.830 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:33.831 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:33.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.827+0000 7fca1cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:33.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.828+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1800163023 0 0) 0x7fca20145e90 con 0x7fca200b38c0 2024-03-29T11:41:33.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.828+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 >> v1:172.21.15.183:6789/0 conn(0x7fca200a9c00 legacy=0x7fca20145780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:41:33.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.828+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 >> v1:172.21.15.155:6789/0 conn(0x7fca200b2800 legacy=0x7fca201e3d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:33.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.829+0000 7fca1cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:41:33.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.830+0000 7fca1cff9640 1 monclient: found mon.c 2024-03-29T11:41:33.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.830+0000 7fca1cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:33.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.830+0000 7fca1cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:41:33.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.830+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca20146020 con 0x7fca200b38c0 2024-03-29T11:41:33.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.830+0000 7fca1cff9640 10 monclient: _finish_auth 0 2024-03-29T11:41:33.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.830+0000 7fca1cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:41:33.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.830+0000 7fca1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:33.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca251bc640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:41:33.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca251bc640 10 monclient: _renew_subs 2024-03-29T11:41:33.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca251bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:41:33.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca251bc640 1 -- 172.21.15.155:0/4154068785 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca20146270 con 0x7fca200b38c0 2024-03-29T11:41:33.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca251bc640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:33.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca251bc640 10 monclient: _renew_subs 2024-03-29T11:41:33.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca251bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:41:33.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca251bc640 1 -- 172.21.15.155:0/4154068785 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca201ef950 con 0x7fca200b38c0 2024-03-29T11:41:33.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca251bc640 1 librados: init done 2024-03-29T11:41:33.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca180020a0 con 0x7fca200b38c0 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca1cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fca18004d80 con 0x7fca200b38c0 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca1cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.832+0000 7fca1cff9640 10 monclient: dump: 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:33.835 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:33.836 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:33.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.833+0000 7fca1e7fc640 4 set_mon_vals no callback set 2024-03-29T11:41:33.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.835+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fca1801ad50 con 0x7fca200b38c0 2024-03-29T11:41:33.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.835+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fca18084e10 con 0x7fca200b38c0 2024-03-29T11:41:33.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.835+0000 7fca1cff9640 10 client.4520.objecter ms_dispatch 0x7fca20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:33.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.835+0000 7fca1cff9640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:33.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.835+0000 7fca1cff9640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:33.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.835+0000 7fca1cff9640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T11:41:33.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.835+0000 7fca251bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:33.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.836+0000 7fca251bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:33.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.836+0000 7fca251bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:41:33.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.836+0000 7fca251bc640 1 -- 172.21.15.155:0/4154068785 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca00005080 con 0x7fca200b38c0 2024-03-29T11:41:33.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.836+0000 7fca1ffff640 10 monclient: get_auth_request con 0x7fca0c06dc20 auth_method 0 2024-03-29T11:41:33.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.836+0000 7fca1cff9640 10 client.4520.objecter ms_handle_connect 0x7fca0c06dc20 2024-03-29T11:41:33.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.844+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca18050520 con 0x7fca200b38c0 2024-03-29T11:41:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.844+0000 7fca1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:33.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:33.844+0000 7fca1cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:34.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.180+0000 7fca251bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:34.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.180+0000 7fca251bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:34.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.180+0000 7fca251bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:41:34.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.180+0000 7fca251bc640 1 -- 172.21.15.155:0/4154068785 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca00005330 con 0x7fca200b38c0 2024-03-29T11:41:34.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.182+0000 7fca1cff9640 1 -- 172.21.15.155:0/4154068785 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fca18052620 con 0x7fca200b38c0 2024-03-29T11:41:34.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.182+0000 7fca1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:34.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.182+0000 7fca1cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:34.184 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:41:34.184 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:34.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.185+0000 7fca251bc640 10 librados: watch_flush enter 2024-03-29T11:41:34.188 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.186+0000 7fca251bc640 10 librados: watch_flush exit 2024-03-29T11:41:34.188 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.186+0000 7fca251bc640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T11:41:34.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.186+0000 7fca251bc640 10 client.4520.objecter successfully canceled tick 2024-03-29T11:41:34.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.186+0000 7fca251bc640 1 -- 172.21.15.155:0/4154068785 >> v1:172.21.15.155:6816/1760814708 conn(0x7fca0c06dc20 legacy=0x7fca0c0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:34.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.186+0000 7fca251bc640 10 monclient: shutdown 2024-03-29T11:41:34.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.186+0000 7fca251bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:34.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.186+0000 7fca251bc640 1 -- 172.21.15.155:0/4154068785 >> v1:172.21.15.155:6790/0 conn(0x7fca200b38c0 legacy=0x7fca201e44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:34.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.187+0000 7fca251bc640 1 -- 172.21.15.155:0/4154068785 shutdown_connections 2024-03-29T11:41:34.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.187+0000 7fca251bc640 1 -- 172.21.15.155:0/4154068785 >> 172.21.15.155:0/4154068785 conn(0x7fca20066a00 msgr2=0x7fca200b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:34.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.190+0000 7fca251bc640 1 -- 172.21.15.155:0/4154068785 shutdown_connections 2024-03-29T11:41:34.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.190+0000 7fca251bc640 1 -- 172.21.15.155:0/4154068785 wait complete. 2024-03-29T11:41:34.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:34.190+0000 7fca251bc640 1 librados: shutdown 2024-03-29T11:41:34.210 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:41.016 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:41.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.165+0000 7ff644c64640 10 monclient: get_monmap_and_config 2024-03-29T11:41:41.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.165+0000 7ff644c64640 10 monclient: build_initial_monmap 2024-03-29T11:41:41.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.165+0000 7ff644c64640 10 monclient: monmap: 2024-03-29T11:41:41.168 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:41.168 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:41.168 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:41:41.166084+0000 2024-03-29T11:41:41.168 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:41:41.166084+0000 2024-03-29T11:41:41.168 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:41.168 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:41.168 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:41.168 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:41:41.168 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:41:41.168 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:41.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.167+0000 7ff644c64640 1 Processor -- start 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.167+0000 7ff644c64640 1 -- start start 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.167+0000 7ff644c64640 10 monclient: init 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.168+0000 7ff644c64640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.168+0000 7ff644c64640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.168+0000 7ff644c64640 10 monclient(hunting): picked mon.noname-c con 0x7ff6401457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.168+0000 7ff644c64640 10 monclient(hunting): picked mon.noname-a con 0x7ff64013f650 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.168+0000 7ff644c64640 10 monclient(hunting): picked mon.noname-b con 0x7ff6400a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.168+0000 7ff644c64640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400aaef0 con 0x7ff64013f650 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.168+0000 7ff644c64640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff640145bd0 con 0x7ff6401457f0 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.168+0000 7ff644c64640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff640145eb0 con 0x7ff6400a9d70 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.168+0000 7ff644c64640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:41.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.168+0000 7ff644c64640 10 monclient(hunting): authenticate will time out at 1283.294393s 2024-03-29T11:41:41.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.172+0000 7ff63ed9d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ff6400a9d70 0x7ff64014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:38812/0 (socket says 172.21.15.155:38812) 2024-03-29T11:41:41.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.172+0000 7ff63ed9d640 1 -- 172.21.15.155:0/144132743 learned_addr learned my addr 172.21.15.155:0/144132743 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:41.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.173+0000 7ff63d59a640 1 -- 172.21.15.155:0/144132743 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053060902 0 0) 0x7ff640145eb0 con 0x7ff6400a9d70 2024-03-29T11:41:41.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.173+0000 7ff63d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:41.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.173+0000 7ff63d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:41.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.173+0000 7ff63d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:41.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.173+0000 7ff63d59a640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:41:41.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.173+0000 7ff63d59a640 1 -- 172.21.15.155:0/144132743 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff628003520 con 0x7ff6400a9d70 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 1 -- 172.21.15.155:0/144132743 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4146853593 0 0) 0x7ff628003520 con 0x7ff6400a9d70 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 1 -- 172.21.15.155:0/144132743 >> v1:172.21.15.183:6789/0 conn(0x7ff6401457f0 legacy=0x7ff64013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 1 -- 172.21.15.155:0/144132743 >> v1:172.21.15.155:6789/0 conn(0x7ff64013f650 legacy=0x7ff6400b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 1 monclient: found mon.noname-b 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 1 -- 172.21.15.155:0/144132743 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff640146340 con 0x7ff6400a9d70 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 10 monclient: _finish_auth 0 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:41.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 1 -- 172.21.15.155:0/144132743 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff638001a10 con 0x7ff6400a9d70 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 10 monclient: dump: 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff644c64640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:41:41.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff644c64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 1 -- 172.21.15.155:0/144132743 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff638001be0 con 0x7ff6400a9d70 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 1 -- 172.21.15.155:0/144132743 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff638001ff0 con 0x7ff6400a9d70 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff63d59a640 10 monclient: dump: 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:41.180 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:41.181 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:41.181 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:41.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff644c64640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:41.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.174+0000 7ff644c64640 4 set_mon_vals no callback set 2024-03-29T11:41:41.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.175+0000 7ff644c64640 10 monclient: shutdown 2024-03-29T11:41:41.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.175+0000 7ff644c64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:41.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.175+0000 7ff644c64640 1 -- 172.21.15.155:0/144132743 >> v1:172.21.15.155:6790/0 conn(0x7ff6400a9d70 legacy=0x7ff64014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.176+0000 7ff644c64640 1 -- 172.21.15.155:0/144132743 shutdown_connections 2024-03-29T11:41:41.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.176+0000 7ff644c64640 1 -- 172.21.15.155:0/144132743 >> 172.21.15.155:0/144132743 conn(0x7ff640066a00 msgr2=0x7ff640066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:41.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.179+0000 7ff644c64640 1 -- 172.21.15.155:0/144132743 shutdown_connections 2024-03-29T11:41:41.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.179+0000 7ff644c64640 1 -- 172.21.15.155:0/144132743 wait complete. 2024-03-29T11:41:41.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.180+0000 7ff644c64640 10 monclient: build_initial_monmap 2024-03-29T11:41:41.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.180+0000 7ff644c64640 10 monclient: monmap: 2024-03-29T11:41:41.183 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.181+0000 7ff644c64640 1 librados: starting msgr at 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.181+0000 7ff644c64640 1 librados: starting objecter 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.181+0000 7ff644c64640 1 Processor -- start 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.181+0000 7ff644c64640 1 -- start start 2024-03-29T11:41:41.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.181+0000 7ff644c64640 1 librados: setting wanted keys 2024-03-29T11:41:41.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.181+0000 7ff644c64640 1 librados: calling monclient init 2024-03-29T11:41:41.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.181+0000 7ff644c64640 10 monclient: init 2024-03-29T11:41:41.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.182+0000 7ff644c64640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:41.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.182+0000 7ff644c64640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:41.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.183+0000 7ff644c64640 10 monclient(hunting): picked mon.noname-b con 0x7ff6400a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:41.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.183+0000 7ff644c64640 10 monclient(hunting): picked mon.noname-a con 0x7ff64013f650 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:41.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.183+0000 7ff644c64640 10 monclient(hunting): picked mon.noname-c con 0x7ff6401457f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:41.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.183+0000 7ff644c64640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff640133770 con 0x7ff64013f650 2024-03-29T11:41:41.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.183+0000 7ff644c64640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400ac660 con 0x7ff6400a9d70 2024-03-29T11:41:41.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.183+0000 7ff644c64640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400ac940 con 0x7ff6401457f0 2024-03-29T11:41:41.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.183+0000 7ff644c64640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:41.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.183+0000 7ff644c64640 10 monclient(hunting): authenticate will time out at 1283.309848s 2024-03-29T11:41:41.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.185+0000 7ff63e59c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff6400a9d70 0x7ff640070d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:43725/0 (socket says 172.21.15.155:43725) 2024-03-29T11:41:41.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.185+0000 7ff63e59c640 1 -- 172.21.15.155:0/4204265323 learned_addr learned my addr 172.21.15.155:0/4204265323 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:41.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.185+0000 7ff62effd640 10 client.?.objecter ms_handle_connect 0x7ff6400a9d70 2024-03-29T11:41:41.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.185+0000 7ff62effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:41.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.185+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804611189 0 0) 0x7ff6400ac660 con 0x7ff6400a9d70 2024-03-29T11:41:41.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:41.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:41.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:41.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:41:41.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff620004eb0 con 0x7ff6400a9d70 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 10 client.?.objecter ms_handle_connect 0x7ff6401457f0 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705102499 0 0) 0x7ff6400ac940 con 0x7ff6401457f0 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.186+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6400ac660 con 0x7ff6401457f0 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4146669204 0 0) 0x7ff620004eb0 con 0x7ff6400a9d70 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6400ac940 con 0x7ff6400a9d70 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff634003fe0 con 0x7ff6400a9d70 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:41.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 10 monclient(hunting): dump: 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3295687782 0 0) 0x7ff6400ac940 con 0x7ff6400a9d70 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 >> v1:172.21.15.155:6790/0 conn(0x7ff6401457f0 legacy=0x7ff6401e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 >> v1:172.21.15.155:6789/0 conn(0x7ff64013f650 legacy=0x7ff640071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:41.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 1 monclient: found mon.b 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64013c4c0 con 0x7ff6400a9d70 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 10 monclient: _finish_auth 0 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff62effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff644c64640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff644c64640 10 monclient: _renew_subs 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff644c64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.187+0000 7ff644c64640 1 -- 172.21.15.155:0/4204265323 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64013c7a0 con 0x7ff6400a9d70 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.188+0000 7ff644c64640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.188+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff634000ed0 con 0x7ff6400a9d70 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.188+0000 7ff62effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:41.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.188+0000 7ff62ffff640 4 set_mon_vals no callback set 2024-03-29T11:41:41.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.189+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff634001220 con 0x7ff6400a9d70 2024-03-29T11:41:41.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.189+0000 7ff644c64640 10 monclient: _renew_subs 2024-03-29T11:41:41.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.189+0000 7ff644c64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:41.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.189+0000 7ff644c64640 1 -- 172.21.15.155:0/4204265323 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6400ab7e0 con 0x7ff6400a9d70 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.189+0000 7ff644c64640 1 librados: init done 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.189+0000 7ff62effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.189+0000 7ff62effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.189+0000 7ff62effd640 10 monclient: dump: 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:41.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.190+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7ff634001440 con 0x7ff6400a9d70 2024-03-29T11:41:41.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.192+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7ff634084a00 con 0x7ff6400a9d70 2024-03-29T11:41:41.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.192+0000 7ff62effd640 10 client.4447.objecter ms_dispatch 0x7ff640000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:41.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.192+0000 7ff63dd9b640 10 monclient: get_auth_request con 0x7ff62006dd00 auth_method 0 2024-03-29T11:41:41.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.192+0000 7ff62effd640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:41.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.192+0000 7ff62effd640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:41.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.192+0000 7ff62effd640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T11:41:41.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.193+0000 7ff62effd640 10 client.4447.objecter ms_handle_connect 0x7ff62006dd00 2024-03-29T11:41:41.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.194+0000 7ff644c64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:41.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.194+0000 7ff644c64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:41.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.194+0000 7ff644c64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:41.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.194+0000 7ff644c64640 1 -- 172.21.15.155:0/4204265323 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64006f3b0 con 0x7ff6400a9d70 2024-03-29T11:41:41.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.205+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff634050300 con 0x7ff6400a9d70 2024-03-29T11:41:41.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.205+0000 7ff62effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:41.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.205+0000 7ff62effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:41.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.531+0000 7ff644c64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:41.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.531+0000 7ff644c64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:41.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.531+0000 7ff644c64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:41.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.531+0000 7ff644c64640 1 -- 172.21.15.155:0/4204265323 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff640075f10 con 0x7ff6400a9d70 2024-03-29T11:41:41.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.535+0000 7ff62effd640 1 -- 172.21.15.155:0/4204265323 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7ff634052230 con 0x7ff6400a9d70 2024-03-29T11:41:41.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.535+0000 7ff62effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:41.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.535+0000 7ff62effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:41.537 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:41:41.537 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:41.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.539+0000 7ff644c64640 10 librados: watch_flush enter 2024-03-29T11:41:41.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.539+0000 7ff644c64640 10 librados: watch_flush exit 2024-03-29T11:41:41.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.539+0000 7ff644c64640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T11:41:41.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.539+0000 7ff644c64640 10 client.4447.objecter successfully canceled tick 2024-03-29T11:41:41.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.539+0000 7ff644c64640 1 -- 172.21.15.155:0/4204265323 >> v1:172.21.15.155:6816/1760814708 conn(0x7ff62006dd00 legacy=0x7ff6200701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.539+0000 7ff644c64640 10 monclient: shutdown 2024-03-29T11:41:41.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.539+0000 7ff644c64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:41.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.539+0000 7ff644c64640 1 -- 172.21.15.155:0/4204265323 >> v1:172.21.15.183:6789/0 conn(0x7ff6400a9d70 legacy=0x7ff640070d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.540+0000 7ff644c64640 1 -- 172.21.15.155:0/4204265323 shutdown_connections 2024-03-29T11:41:41.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.540+0000 7ff644c64640 1 -- 172.21.15.155:0/4204265323 >> 172.21.15.155:0/4204265323 conn(0x7ff640066a00 msgr2=0x7ff6400b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:41.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.540+0000 7ff644c64640 1 -- 172.21.15.155:0/4204265323 shutdown_connections 2024-03-29T11:41:41.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.540+0000 7ff644c64640 1 -- 172.21.15.155:0/4204265323 wait complete. 2024-03-29T11:41:41.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:41.540+0000 7ff644c64640 1 librados: shutdown 2024-03-29T11:41:41.561 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:48.367 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.524+0000 7f620e692640 10 monclient: get_monmap_and_config 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.524+0000 7f620e692640 10 monclient: build_initial_monmap 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.524+0000 7f620e692640 10 monclient: monmap: 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:41:48.526386+0000 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:41:48.526386+0000 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:41:48.528 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:41:48.529 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:48.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.527+0000 7f620e692640 1 Processor -- start 2024-03-29T11:41:48.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.527+0000 7f620e692640 1 -- start start 2024-03-29T11:41:48.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.527+0000 7f620e692640 10 monclient: init 2024-03-29T11:41:48.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.528+0000 7f620e692640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:48.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.528+0000 7f620e692640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:48.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.528+0000 7f620e692640 10 monclient(hunting): picked mon.noname-a con 0x7f62000389e0 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:48.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.528+0000 7f620e692640 10 monclient(hunting): picked mon.noname-c con 0x7f6200038dc0 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:48.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.528+0000 7f620e692640 10 monclient(hunting): picked mon.noname-b con 0x7f6200030660 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:48.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.528+0000 7f620e692640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62000c54c0 con 0x7f62000389e0 2024-03-29T11:41:48.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.528+0000 7f620e692640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6200036430 con 0x7f6200038dc0 2024-03-29T11:41:48.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.528+0000 7f620e692640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6200039840 con 0x7f6200030660 2024-03-29T11:41:48.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.528+0000 7f620e692640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:48.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.528+0000 7f620e692640 10 monclient(hunting): authenticate will time out at 1290.655711s 2024-03-29T11:41:48.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.531+0000 7f620de91640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f6200030660 0x7f62000d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:53610/0 (socket says 172.21.15.155:53610) 2024-03-29T11:41:48.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.531+0000 7f620de91640 1 -- 172.21.15.155:0/2523467700 learned_addr learned my addr 172.21.15.155:0/2523467700 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:48.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.532+0000 7f61fffff640 1 -- 172.21.15.155:0/2523467700 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184966197 0 0) 0x7f62000c54c0 con 0x7f62000389e0 2024-03-29T11:41:48.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.532+0000 7f61fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:48.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.532+0000 7f61fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:48.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.532+0000 7f61fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:48.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.532+0000 7f61fffff640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:41:48.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.532+0000 7f61fffff640 1 -- 172.21.15.155:0/2523467700 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f4003520 con 0x7f62000389e0 2024-03-29T11:41:48.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.532+0000 7f61fffff640 1 -- 172.21.15.155:0/2523467700 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3710649569 0 0) 0x7f61f4003520 con 0x7f62000389e0 2024-03-29T11:41:48.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.532+0000 7f61fffff640 1 -- 172.21.15.155:0/2523467700 >> v1:172.21.15.155:6790/0 conn(0x7f6200030660 legacy=0x7f62000d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:48.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.532+0000 7f61fffff640 1 -- 172.21.15.155:0/2523467700 >> v1:172.21.15.183:6789/0 conn(0x7f6200038dc0 legacy=0x7f62000cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:48.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:48.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 1 monclient: found mon.noname-a 2024-03-29T11:41:48.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:48.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:41:48.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 1 -- 172.21.15.155:0/2523467700 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6200035690 con 0x7f62000389e0 2024-03-29T11:41:48.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 10 monclient: _finish_auth 0 2024-03-29T11:41:48.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:48.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:48.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f620e692640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:41:48.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f620e692640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 1 -- 172.21.15.155:0/2523467700 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f61f8001690 con 0x7f62000389e0 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 10 monclient: dump: 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:48.544 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f620e692640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 1 -- 172.21.15.155:0/2523467700 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61f8001920 con 0x7f62000389e0 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f61fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f620e692640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.535+0000 7f620e692640 4 set_mon_vals no callback set 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.536+0000 7f61fffff640 1 -- 172.21.15.155:0/2523467700 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f61f8001cb0 con 0x7f62000389e0 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.536+0000 7f61fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.536+0000 7f61fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.536+0000 7f61fffff640 10 monclient: dump: 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:48.545 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.536+0000 7f620e692640 10 monclient: shutdown 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.536+0000 7f620e692640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.536+0000 7f620e692640 1 -- 172.21.15.155:0/2523467700 >> v1:172.21.15.155:6789/0 conn(0x7f62000389e0 legacy=0x7f6200033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.538+0000 7f620e692640 1 -- 172.21.15.155:0/2523467700 shutdown_connections 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.538+0000 7f620e692640 1 -- 172.21.15.155:0/2523467700 >> 172.21.15.155:0/2523467700 conn(0x7f6200005c00 msgr2=0x7f6200006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.538+0000 7f620e692640 1 -- 172.21.15.155:0/2523467700 shutdown_connections 2024-03-29T11:41:48.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.538+0000 7f620e692640 1 -- 172.21.15.155:0/2523467700 wait complete. 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.539+0000 7f620e692640 10 monclient: build_initial_monmap 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.539+0000 7f620e692640 10 monclient: monmap: 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:48.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.540+0000 7f620e692640 1 librados: starting msgr at 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.540+0000 7f620e692640 1 librados: starting objecter 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.540+0000 7f620e692640 1 Processor -- start 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.540+0000 7f620e692640 1 -- start start 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.540+0000 7f620e692640 1 librados: setting wanted keys 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.540+0000 7f620e692640 1 librados: calling monclient init 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.540+0000 7f620e692640 10 monclient: init 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.541+0000 7f620e692640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.541+0000 7f620e692640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.541+0000 7f620e692640 10 monclient(hunting): picked mon.noname-a con 0x7f6200030660 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.541+0000 7f620e692640 10 monclient(hunting): picked mon.noname-c con 0x7f62000389e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.541+0000 7f620e692640 10 monclient(hunting): picked mon.noname-b con 0x7f62000be4b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.541+0000 7f620e692640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62000bf6a0 con 0x7f6200030660 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.541+0000 7f620e692640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62000bf980 con 0x7f62000be4b0 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.541+0000 7f620e692640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620000d530 con 0x7f62000389e0 2024-03-29T11:41:48.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.541+0000 7f620e692640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:48.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.541+0000 7f620e692640 10 monclient(hunting): authenticate will time out at 1290.668235s 2024-03-29T11:41:48.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.548+0000 7f620de91640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f62000be4b0 0x7f620000b010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:53437/0 (socket says 172.21.15.155:53437) 2024-03-29T11:41:48.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.548+0000 7f620de91640 1 -- 172.21.15.155:0/420049759 learned_addr learned my addr 172.21.15.155:0/420049759 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:48.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.548+0000 7f61fdffb640 10 client.?.objecter ms_handle_connect 0x7f62000be4b0 2024-03-29T11:41:48.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.548+0000 7f61fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:48.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.549+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478619886 0 0) 0x7f62000bf980 con 0x7f62000be4b0 2024-03-29T11:41:48.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.549+0000 7f61fdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:48.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.549+0000 7f61fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:48.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.549+0000 7f61fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:48.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.549+0000 7f61fdffb640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:41:48.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.549+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ec004eb0 con 0x7f62000be4b0 2024-03-29T11:41:48.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.549+0000 7f61fdffb640 10 client.?.objecter ms_handle_connect 0x7f62000389e0 2024-03-29T11:41:48.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.549+0000 7f61fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3918770146 0 0) 0x7f61ec004eb0 con 0x7f62000be4b0 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62000bf980 con 0x7f62000be4b0 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f620804fdd0 con 0x7f62000be4b0 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient(hunting): dump: 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:48.553 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:48.554 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:48.554 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:48.554 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:48.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:48.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005107292 0 0) 0x7f620000d530 con 0x7f62000389e0 2024-03-29T11:41:48.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:48.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:48.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:48.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:41:48.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ec004eb0 con 0x7f62000389e0 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 965192143 0 0) 0x7f62000bf980 con 0x7f62000be4b0 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 >> v1:172.21.15.155:6790/0 conn(0x7f62000389e0 legacy=0x7f62000c04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 >> v1:172.21.15.155:6789/0 conn(0x7f6200030660 legacy=0x7f62000bfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 1 monclient: found mon.b 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62001783e0 con 0x7f62000be4b0 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f61fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f620e692640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:41:48.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f620e692640 10 monclient: _renew_subs 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f620e692640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.550+0000 7f620e692640 1 -- 172.21.15.155:0/420049759 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62001753b0 con 0x7f62000be4b0 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.551+0000 7f620e692640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.551+0000 7f620e692640 10 monclient: _renew_subs 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.551+0000 7f620e692640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.551+0000 7f620e692640 1 -- 172.21.15.155:0/420049759 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6200175840 con 0x7f62000be4b0 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.551+0000 7f620e692640 1 librados: init done 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.551+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f620804f400 con 0x7f62000be4b0 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.551+0000 7f61fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.551+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6208075710 con 0x7f62000be4b0 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.551+0000 7f61fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.551+0000 7f61fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.551+0000 7f61fdffb640 10 monclient: dump: 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:48.556 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:48.557 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:48.557 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:48.557 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:48.557 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:48.557 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:48.557 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:48.557 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:48.557 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:48.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.554+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f6208067bf0 con 0x7f62000be4b0 2024-03-29T11:41:48.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.555+0000 7f61ff7fe640 4 set_mon_vals no callback set 2024-03-29T11:41:48.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.555+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f62080ff5a0 con 0x7f62000be4b0 2024-03-29T11:41:48.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.555+0000 7f61fdffb640 10 client.4450.objecter ms_dispatch 0x7f6208000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:48.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.555+0000 7f61fdffb640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:48.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.555+0000 7f61fdffb640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:48.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.555+0000 7f61fdffb640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-29T11:41:48.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.556+0000 7f620e692640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:48.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.556+0000 7f620e692640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:48.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.556+0000 7f620e692640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:48.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.556+0000 7f620e692640 1 -- 172.21.15.155:0/420049759 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e8005080 con 0x7f62000be4b0 2024-03-29T11:41:48.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.556+0000 7f620d690640 10 monclient: get_auth_request con 0x7f61ec06db90 auth_method 0 2024-03-29T11:41:48.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.556+0000 7f61fdffb640 10 client.4450.objecter ms_handle_connect 0x7f61ec06db90 2024-03-29T11:41:48.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.564+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62080cae80 con 0x7f62000be4b0 2024-03-29T11:41:48.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.564+0000 7f61fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:48.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.564+0000 7f61fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:48.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.916+0000 7f620e692640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:48.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.916+0000 7f620e692640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:48.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.916+0000 7f620e692640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:41:48.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.916+0000 7f620e692640 1 -- 172.21.15.155:0/420049759 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61e8005330 con 0x7f62000be4b0 2024-03-29T11:41:48.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.917+0000 7f61fdffb640 1 -- 172.21.15.155:0/420049759 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f62080ccdb0 con 0x7f62000be4b0 2024-03-29T11:41:48.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.917+0000 7f61fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:48.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.917+0000 7f61fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:48.920 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:41:48.920 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:48.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.921+0000 7f620e692640 10 librados: watch_flush enter 2024-03-29T11:41:48.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.922+0000 7f620e692640 10 librados: watch_flush exit 2024-03-29T11:41:48.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.922+0000 7f620e692640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-29T11:41:48.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.922+0000 7f620e692640 10 client.4450.objecter successfully canceled tick 2024-03-29T11:41:48.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.923+0000 7f620e692640 1 -- 172.21.15.155:0/420049759 >> v1:172.21.15.155:6816/1760814708 conn(0x7f61ec06db90 legacy=0x7f61ec070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:48.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.923+0000 7f620e692640 10 monclient: shutdown 2024-03-29T11:41:48.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.923+0000 7f620e692640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:48.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.923+0000 7f620e692640 1 -- 172.21.15.155:0/420049759 >> v1:172.21.15.183:6789/0 conn(0x7f62000be4b0 legacy=0x7f620000b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:48.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.923+0000 7f620e692640 1 -- 172.21.15.155:0/420049759 shutdown_connections 2024-03-29T11:41:48.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.923+0000 7f620e692640 1 -- 172.21.15.155:0/420049759 >> 172.21.15.155:0/420049759 conn(0x7f6200005c00 msgr2=0x7f6200034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:48.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.923+0000 7f620e692640 1 -- 172.21.15.155:0/420049759 shutdown_connections 2024-03-29T11:41:48.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.924+0000 7f620e692640 1 -- 172.21.15.155:0/420049759 wait complete. 2024-03-29T11:41:48.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:48.924+0000 7f620e692640 1 librados: shutdown 2024-03-29T11:41:48.952 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:55.758 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:55.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.906+0000 7f40030ba640 10 monclient: get_monmap_and_config 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.906+0000 7f40030ba640 10 monclient: build_initial_monmap 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.906+0000 7f40030ba640 10 monclient: monmap: 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:41:55.908171+0000 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:41:55.908171+0000 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:41:55.910 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:55.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.907+0000 7f40030ba640 1 Processor -- start 2024-03-29T11:41:55.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.907+0000 7f40030ba640 1 -- start start 2024-03-29T11:41:55.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.907+0000 7f40030ba640 10 monclient: init 2024-03-29T11:41:55.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.908+0000 7f40030ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:55.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.908+0000 7f40030ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:55.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.908+0000 7f40030ba640 10 monclient(hunting): picked mon.noname-c con 0x7f3ffc1457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:55.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.909+0000 7f40030ba640 10 monclient(hunting): picked mon.noname-a con 0x7f3ffc13f650 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:55.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.909+0000 7f40030ba640 10 monclient(hunting): picked mon.noname-b con 0x7f3ffc0a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:55.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.909+0000 7f40030ba640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc0aaef0 con 0x7f3ffc13f650 2024-03-29T11:41:55.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.909+0000 7f40030ba640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc145bd0 con 0x7f3ffc1457f0 2024-03-29T11:41:55.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.909+0000 7f40030ba640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc145eb0 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.909+0000 7f40030ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:55.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.909+0000 7f40030ba640 10 monclient(hunting): authenticate will time out at 1298.035939s 2024-03-29T11:41:55.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.912+0000 7f4001657640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f3ffc0a9d70 0x7f3ffc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:57967/0 (socket says 172.21.15.155:57967) 2024-03-29T11:41:55.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.912+0000 7f4001657640 1 -- 172.21.15.155:0/996602783 learned_addr learned my addr 172.21.15.155:0/996602783 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:55.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 1 -- 172.21.15.155:0/996602783 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922285480 0 0) 0x7f3ffc145eb0 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:55.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:55.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:55.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:41:55.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 1 -- 172.21.15.155:0/996602783 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe4003520 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 1 -- 172.21.15.155:0/996602783 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 869157851 0 0) 0x7f3fe4003520 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 1 -- 172.21.15.155:0/996602783 >> v1:172.21.15.183:6789/0 conn(0x7f3ffc1457f0 legacy=0x7f3ffc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:55.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 1 -- 172.21.15.155:0/996602783 >> v1:172.21.15.155:6789/0 conn(0x7f3ffc13f650 legacy=0x7f3ffc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:55.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 1 -- 172.21.15.155:0/996602783 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc146340 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 1 -- 172.21.15.155:0/996602783 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3ff4001a10 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.913+0000 7f3ffb7fe640 10 monclient: dump: 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:55.921 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.914+0000 7f40030ba640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.914+0000 7f40030ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.914+0000 7f3ffb7fe640 1 -- 172.21.15.155:0/996602783 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ff4001be0 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.914+0000 7f3ffb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.914+0000 7f3ffb7fe640 1 -- 172.21.15.155:0/996602783 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3ff4001ff0 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.914+0000 7f3ffb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.914+0000 7f3ffb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.914+0000 7f3ffb7fe640 10 monclient: dump: 2024-03-29T11:41:55.922 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.914+0000 7f40030ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.914+0000 7f40030ba640 4 set_mon_vals no callback set 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.915+0000 7f40030ba640 10 monclient: shutdown 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.915+0000 7f40030ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.915+0000 7f40030ba640 1 -- 172.21.15.155:0/996602783 >> v1:172.21.15.155:6790/0 conn(0x7f3ffc0a9d70 legacy=0x7f3ffc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:55.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.916+0000 7f40030ba640 1 -- 172.21.15.155:0/996602783 shutdown_connections 2024-03-29T11:41:55.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.916+0000 7f40030ba640 1 -- 172.21.15.155:0/996602783 >> 172.21.15.155:0/996602783 conn(0x7f3ffc066a00 msgr2=0x7f3ffc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:55.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.916+0000 7f40030ba640 1 -- 172.21.15.155:0/996602783 shutdown_connections 2024-03-29T11:41:55.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.916+0000 7f40030ba640 1 -- 172.21.15.155:0/996602783 wait complete. 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.917+0000 7f40030ba640 10 monclient: build_initial_monmap 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.917+0000 7f40030ba640 10 monclient: monmap: 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:55.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.917+0000 7f40030ba640 1 librados: starting msgr at 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.917+0000 7f40030ba640 1 librados: starting objecter 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.917+0000 7f40030ba640 1 Processor -- start 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.918+0000 7f40030ba640 1 -- start start 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.918+0000 7f40030ba640 1 librados: setting wanted keys 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.918+0000 7f40030ba640 1 librados: calling monclient init 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.918+0000 7f40030ba640 10 monclient: init 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.919+0000 7f40030ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.919+0000 7f40030ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.919+0000 7f40030ba640 10 monclient(hunting): picked mon.noname-a con 0x7f3ffc0a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.919+0000 7f40030ba640 10 monclient(hunting): picked mon.noname-b con 0x7f3ffc13f650 addr v1:172.21.15.183:6789/0 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.919+0000 7f40030ba640 10 monclient(hunting): picked mon.noname-c con 0x7f3ffc1457f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.919+0000 7f40030ba640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc1421e0 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.919+0000 7f40030ba640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc137b10 con 0x7f3ffc13f650 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.919+0000 7f40030ba640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc071da0 con 0x7f3ffc1457f0 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.919+0000 7f40030ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.919+0000 7f40030ba640 10 monclient(hunting): authenticate will time out at 1298.046270s 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.920+0000 7f4000e56640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f3ffc0a9d70 0x7f3ffc142780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:47036/0 (socket says 172.21.15.155:47036) 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.920+0000 7f4000e56640 1 -- 172.21.15.155:0/2227903135 learned_addr learned my addr 172.21.15.155:0/2227903135 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.920+0000 7f3ff97fa640 10 client.?.objecter ms_handle_connect 0x7f3ffc0a9d70 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.920+0000 7f3ff97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.920+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919575069 0 0) 0x7f3fec003820 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.920+0000 7f3ff97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.920+0000 7f3ff97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.920+0000 7f3ff97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.920+0000 7f3ff97fa640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.920+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe8004eb0 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.921+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2881120426 0 0) 0x7f3fe8004eb0 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.921+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fec003820 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.921+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3fec003fd0 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.921+0000 7f3ff97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.921+0000 7f3ff97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.921+0000 7f3ff97fa640 10 monclient(hunting): dump: 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:55.933 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.921+0000 7f3ff97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.923+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 601168170 0 0) 0x7f3fec003820 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.923+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 >> v1:172.21.15.155:6790/0 conn(0x7f3ffc1457f0 legacy=0x7f3ffc071690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.923+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 >> v1:172.21.15.183:6789/0 conn(0x7f3ffc13f650 legacy=0x7f3ffc1418d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f3ff97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:41:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f3ff97fa640 1 monclient: found mon.a 2024-03-29T11:41:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f3ff97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f3ff97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc071f30 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f3ff97fa640 10 monclient: _finish_auth 0 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f3ff97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f3ff97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f40030ba640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f40030ba640 10 monclient: _renew_subs 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f40030ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f40030ba640 1 -- 172.21.15.155:0/2227903135 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ffc144790 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f40030ba640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f40030ba640 10 monclient: _renew_subs 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f40030ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f40030ba640 1 -- 172.21.15.155:0/2227903135 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ffc1db790 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.924+0000 7f40030ba640 1 librados: init done 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.925+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fec004130 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.925+0000 7f3ff97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.925+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3fec001170 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.925+0000 7f3ff97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.925+0000 7f3ff97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.925+0000 7f3ff97fa640 10 monclient: dump: 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:41:55.936 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:41:55.937 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:41:55.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.925+0000 7f3ffa7fc640 4 set_mon_vals no callback set 2024-03-29T11:41:55.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.926+0000 7f3fe2ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:55.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.926+0000 7f3fe2ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:55.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.926+0000 7f3fe2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:55.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.926+0000 7f3fe2ffd640 1 -- 172.21.15.155:0/2227903135 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd4005080 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.928+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f3fec0013d0 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.929+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f3fec0849e0 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.929+0000 7f3ff97fa640 10 client.4476.objecter ms_dispatch 0x7f3ffc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.929+0000 7f3ff97fa640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.929+0000 7f3ff97fa640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.929+0000 7f3ff97fa640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-29T11:41:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.930+0000 7f3ffbfff640 10 monclient: get_auth_request con 0x7f3fe806dbc0 auth_method 0 2024-03-29T11:41:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.931+0000 7f3ff97fa640 10 client.4476.objecter ms_handle_connect 0x7f3fe806dbc0 2024-03-29T11:41:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.935+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3fec0502c0 con 0x7f3ffc0a9d70 2024-03-29T11:41:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.935+0000 7f3ff97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:55.935+0000 7f3ff97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:56.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.266+0000 7f3fe2ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:56.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.266+0000 7f3fe2ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:56.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.266+0000 7f3fe2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:41:56.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.266+0000 7f3fe2ffd640 1 -- 172.21.15.155:0/2227903135 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fd4005330 con 0x7f3ffc0a9d70 2024-03-29T11:41:56.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.267+0000 7f3ff97fa640 1 -- 172.21.15.155:0/2227903135 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f3fec0521f0 con 0x7f3ffc0a9d70 2024-03-29T11:41:56.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.267+0000 7f3ff97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:56.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.267+0000 7f3ff97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:56.272 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:41:56.272 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:56.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.271+0000 7f3fe2ffd640 10 librados: watch_flush enter 2024-03-29T11:41:56.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.271+0000 7f3fe2ffd640 10 librados: watch_flush exit 2024-03-29T11:41:56.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.271+0000 7f3fe2ffd640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-29T11:41:56.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.271+0000 7f3fe2ffd640 10 client.4476.objecter successfully canceled tick 2024-03-29T11:41:56.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.271+0000 7f3fe2ffd640 1 -- 172.21.15.155:0/2227903135 >> v1:172.21.15.155:6816/1760814708 conn(0x7f3fe806dbc0 legacy=0x7f3fe8070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:56.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.271+0000 7f3fe2ffd640 10 monclient: shutdown 2024-03-29T11:41:56.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.271+0000 7f3fe2ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:56.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.271+0000 7f3fe2ffd640 1 -- 172.21.15.155:0/2227903135 >> v1:172.21.15.155:6789/0 conn(0x7f3ffc0a9d70 legacy=0x7f3ffc142780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:56.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.272+0000 7f3fe2ffd640 1 -- 172.21.15.155:0/2227903135 shutdown_connections 2024-03-29T11:41:56.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.272+0000 7f3fe2ffd640 1 -- 172.21.15.155:0/2227903135 >> 172.21.15.155:0/2227903135 conn(0x7f3ffc066a00 msgr2=0x7f3ffc13aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:56.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.272+0000 7f3fe2ffd640 1 -- 172.21.15.155:0/2227903135 shutdown_connections 2024-03-29T11:41:56.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.272+0000 7f3fe2ffd640 1 -- 172.21.15.155:0/2227903135 wait complete. 2024-03-29T11:41:56.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:41:56.272+0000 7f3fe2ffd640 1 librados: shutdown 2024-03-29T11:41:56.292 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:03.100 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:03.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.252+0000 7f007ea3e640 10 monclient: get_monmap_and_config 2024-03-29T11:42:03.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.252+0000 7f007ea3e640 10 monclient: build_initial_monmap 2024-03-29T11:42:03.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.252+0000 7f007ea3e640 10 monclient: monmap: 2024-03-29T11:42:03.255 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:03.255 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:03.256 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:42:03.253701+0000 2024-03-29T11:42:03.256 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:42:03.253701+0000 2024-03-29T11:42:03.256 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:03.256 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:03.256 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:03.256 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:42:03.256 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:42:03.256 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:03.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.253+0000 7f007ea3e640 1 Processor -- start 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.253+0000 7f007ea3e640 1 -- start start 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.253+0000 7f007ea3e640 10 monclient: init 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.254+0000 7f007ea3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.254+0000 7f007ea3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.254+0000 7f007ea3e640 10 monclient(hunting): picked mon.noname-a con 0x7f00781422b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.254+0000 7f007ea3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0078142690 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.254+0000 7f007ea3e640 10 monclient(hunting): picked mon.noname-b con 0x7f00780b17d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.254+0000 7f007ea3e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0078142c80 con 0x7f00781422b0 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.254+0000 7f007ea3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00780aae40 con 0x7f0078142690 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.254+0000 7f0077fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f00781422b0 0x7f007813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:51880/0 (socket says 172.21.15.155:51880) 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.254+0000 7f0077fff640 1 -- 172.21.15.155:0/3462296931 learned_addr learned my addr 172.21.15.155:0/3462296931 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:03.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f007ea3e640 1 -- 172.21.15.155:0/3462296931 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007813fbd0 con 0x7f00780b17d0 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f007ea3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 1 -- 172.21.15.155:0/3462296931 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117791052 0 0) 0x7f0078142c80 con 0x7f00781422b0 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f007ea3e640 10 monclient(hunting): authenticate will time out at 1305.382140s 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 1 -- 172.21.15.155:0/3462296931 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0060003520 con 0x7f00781422b0 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 1 -- 172.21.15.155:0/3462296931 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887508382 0 0) 0x7f00780aae40 con 0x7f0078142690 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:42:03.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 1 -- 172.21.15.155:0/3462296931 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0078142c80 con 0x7f0078142690 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 1 -- 172.21.15.155:0/3462296931 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 437438681 0 0) 0x7f0078142c80 con 0x7f0078142690 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 1 -- 172.21.15.155:0/3462296931 >> v1:172.21.15.155:6790/0 conn(0x7f00780b17d0 legacy=0x7f00780b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 1 -- 172.21.15.155:0/3462296931 >> v1:172.21.15.155:6789/0 conn(0x7f00781422b0 legacy=0x7f007813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 1 monclient: found mon.noname-c 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 1 -- 172.21.15.155:0/3462296931 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007814e910 con 0x7f0078142690 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 1 -- 172.21.15.155:0/3462296931 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f006c0015a0 con 0x7f0078142690 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:03.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f0076ffd640 10 monclient: dump: 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f007ea3e640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:42:03.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.255+0000 7f007ea3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.256+0000 7f0076ffd640 1 -- 172.21.15.155:0/3462296931 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f006c001700 con 0x7f0078142690 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.256+0000 7f0076ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.256+0000 7f0076ffd640 1 -- 172.21.15.155:0/3462296931 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f006c001a90 con 0x7f0078142690 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.256+0000 7f0076ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.256+0000 7f0076ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.256+0000 7f0076ffd640 10 monclient: dump: 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:03.265 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:03.266 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:03.266 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:03.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.256+0000 7f007ea3e640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:03.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.256+0000 7f007ea3e640 4 set_mon_vals no callback set 2024-03-29T11:42:03.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.256+0000 7f007ea3e640 10 monclient: shutdown 2024-03-29T11:42:03.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.257+0000 7f007ea3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:03.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.257+0000 7f007ea3e640 1 -- 172.21.15.155:0/3462296931 >> v1:172.21.15.183:6789/0 conn(0x7f0078142690 legacy=0x7f00780b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.257+0000 7f007ea3e640 1 -- 172.21.15.155:0/3462296931 shutdown_connections 2024-03-29T11:42:03.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.257+0000 7f007ea3e640 1 -- 172.21.15.155:0/3462296931 >> 172.21.15.155:0/3462296931 conn(0x7f0078066a00 msgr2=0x7f0078066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:03.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.257+0000 7f007ea3e640 1 -- 172.21.15.155:0/3462296931 shutdown_connections 2024-03-29T11:42:03.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.257+0000 7f007ea3e640 1 -- 172.21.15.155:0/3462296931 wait complete. 2024-03-29T11:42:03.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.258+0000 7f007ea3e640 10 monclient: build_initial_monmap 2024-03-29T11:42:03.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.258+0000 7f007ea3e640 10 monclient: monmap: 2024-03-29T11:42:03.272 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:03.272 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:03.272 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:42:03.272 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:42:03.272 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:03.272 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:03.272 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:03.272 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:42:03.272 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.258+0000 7f007ea3e640 1 librados: starting msgr at 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.259+0000 7f007ea3e640 1 librados: starting objecter 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.259+0000 7f007ea3e640 1 Processor -- start 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.259+0000 7f007ea3e640 1 -- start start 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.259+0000 7f007ea3e640 1 librados: setting wanted keys 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.259+0000 7f007ea3e640 1 librados: calling monclient init 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.259+0000 7f007ea3e640 10 monclient: init 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.260+0000 7f007ea3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.260+0000 7f007ea3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.260+0000 7f007ea3e640 10 monclient(hunting): picked mon.noname-b con 0x7f00780b17d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.260+0000 7f007ea3e640 10 monclient(hunting): picked mon.noname-a con 0x7f00781422b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:03.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.260+0000 7f007ea3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0078142690 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.260+0000 7f007ea3e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007806e9b0 con 0x7f00781422b0 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.260+0000 7f0077fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f00780b17d0 0x7f0078130460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:44552/0 (socket says 172.21.15.155:44552) 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.260+0000 7f0077fff640 1 -- 172.21.15.155:0/122880000 learned_addr learned my addr 172.21.15.155:0/122880000 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.260+0000 7f0074ff9640 10 client.?.objecter ms_handle_connect 0x7f00780b17d0 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.260+0000 7f0074ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.260+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00780ab790 con 0x7f00780b17d0 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00780aba70 con 0x7f0078142690 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f007ea3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f007ea3e640 10 monclient(hunting): authenticate will time out at 1305.387940s 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 client.?.objecter ms_handle_connect 0x7f00781422b0 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 client.?.objecter ms_handle_connect 0x7f0078142690 2024-03-29T11:42:03.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842145947 0 0) 0x7f00780ab790 con 0x7f00780b17d0 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0064004eb0 con 0x7f00780b17d0 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869291322 0 0) 0x7f007806e9b0 con 0x7f00781422b0 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00780ab790 con 0x7f00781422b0 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979143222 0 0) 0x7f00780aba70 con 0x7f0078142690 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:03.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.261+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f007806e9b0 con 0x7f0078142690 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 611729049 0 0) 0x7f0064004eb0 con 0x7f00780b17d0 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00780aba70 con 0x7f00780b17d0 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0070001cd0 con 0x7f00780b17d0 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:03.276 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1371813432 0 0) 0x7f00780ab790 con 0x7f00781422b0 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0064004eb0 con 0x7f00781422b0 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f006c001a50 con 0x7f00781422b0 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:03.277 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2798250349 0 0) 0x7f00780aba70 con 0x7f00780b17d0 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 >> v1:172.21.15.155:6790/0 conn(0x7f0078142690 legacy=0x7f00781e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.262+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 >> v1:172.21.15.155:6789/0 conn(0x7f00781422b0 legacy=0x7f007806ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 1 monclient: found mon.b 2024-03-29T11:42:03.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007812c0b0 con 0x7f00780b17d0 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 10 monclient: _finish_auth 0 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f007ea3e640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f007ea3e640 10 monclient: _renew_subs 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f007ea3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007806f550 con 0x7f00780b17d0 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f007ea3e640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f007ea3e640 10 monclient: _renew_subs 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f007ea3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00781e7b40 con 0x7f00780b17d0 2024-03-29T11:42:03.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f007ea3e640 1 librados: init done 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0070004b60 con 0x7f00780b17d0 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0070004e70 con 0x7f00780b17d0 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f00767fc640 4 set_mon_vals no callback set 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.263+0000 7f0074ff9640 10 monclient: dump: 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:03.280 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.264+0000 7f007ea3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.264+0000 7f007ea3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.264+0000 7f007ea3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.265+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007806f790 con 0x7f00780b17d0 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.265+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f007001ae40 con 0x7f00780b17d0 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.266+0000 7f00777fe640 10 monclient: get_auth_request con 0x7f006406df70 auth_method 0 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.266+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f0070084e20 con 0x7f00780b17d0 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.266+0000 7f0074ff9640 10 client.4456.objecter ms_dispatch 0x7f0078000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.266+0000 7f0074ff9640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.266+0000 7f0074ff9640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.266+0000 7f0074ff9640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-29T11:42:03.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.266+0000 7f0074ff9640 10 client.4456.objecter ms_handle_connect 0x7f006406df70 2024-03-29T11:42:03.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.274+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0070050700 con 0x7f00780b17d0 2024-03-29T11:42:03.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.274+0000 7f0074ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:03.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.274+0000 7f0074ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:03.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.619+0000 7f007ea3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:03.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.619+0000 7f007ea3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:03.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.619+0000 7f007ea3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:03.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.619+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0078075f10 con 0x7f00780b17d0 2024-03-29T11:42:03.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.625+0000 7f0074ff9640 1 -- 172.21.15.155:0/122880000 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f0070052630 con 0x7f00780b17d0 2024-03-29T11:42:03.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.625+0000 7f0074ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:03.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.625+0000 7f0074ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:03.628 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:42:03.628 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:03.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.628+0000 7f007ea3e640 10 librados: watch_flush enter 2024-03-29T11:42:03.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.628+0000 7f007ea3e640 10 librados: watch_flush exit 2024-03-29T11:42:03.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.628+0000 7f007ea3e640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-29T11:42:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.628+0000 7f007ea3e640 10 client.4456.objecter successfully canceled tick 2024-03-29T11:42:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.629+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 >> v1:172.21.15.155:6816/1760814708 conn(0x7f006406df70 legacy=0x7f0064070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.629+0000 7f007ea3e640 10 monclient: shutdown 2024-03-29T11:42:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.629+0000 7f007ea3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.629+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 >> v1:172.21.15.183:6789/0 conn(0x7f00780b17d0 legacy=0x7f0078130460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.629+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 shutdown_connections 2024-03-29T11:42:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.629+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 >> 172.21.15.155:0/122880000 conn(0x7f0078066a00 msgr2=0x7f007813c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.629+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 shutdown_connections 2024-03-29T11:42:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.629+0000 7f007ea3e640 1 -- 172.21.15.155:0/122880000 wait complete. 2024-03-29T11:42:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:03.629+0000 7f007ea3e640 1 librados: shutdown 2024-03-29T11:42:03.648 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:10.455 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.618+0000 7ff5071e8640 10 monclient: get_monmap_and_config 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.618+0000 7ff5071e8640 10 monclient: build_initial_monmap 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.618+0000 7ff5071e8640 10 monclient: monmap: 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:42:10.619846+0000 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:42:10.619846+0000 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:42:10.622 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:10.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.621+0000 7ff5071e8640 1 Processor -- start 2024-03-29T11:42:10.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.621+0000 7ff5071e8640 1 -- start start 2024-03-29T11:42:10.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.621+0000 7ff5071e8640 10 monclient: init 2024-03-29T11:42:10.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.622+0000 7ff5071e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:10.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.622+0000 7ff5071e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:10.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.622+0000 7ff5071e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff5001457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:10.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.622+0000 7ff5071e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff50013f650 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:10.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.622+0000 7ff5071e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff5000a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:10.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.622+0000 7ff5071e8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5000aaef0 con 0x7ff50013f650 2024-03-29T11:42:10.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.622+0000 7ff5071e8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff500145bd0 con 0x7ff5001457f0 2024-03-29T11:42:10.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.622+0000 7ff5071e8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff500145eb0 con 0x7ff5000a9d70 2024-03-29T11:42:10.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.622+0000 7ff5071e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:10.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.622+0000 7ff5071e8640 10 monclient(hunting): authenticate will time out at 1312.749427s 2024-03-29T11:42:10.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.630+0000 7ff4fffff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff50013f650 0x7ff5000b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:42205/0 (socket says 172.21.15.155:42205) 2024-03-29T11:42:10.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.630+0000 7ff4fffff640 1 -- 172.21.15.155:0/3847616724 learned_addr learned my addr 172.21.15.155:0/3847616724 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.633+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761490184 0 0) 0x7ff500145bd0 con 0x7ff5001457f0 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.633+0000 7ff4ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.633+0000 7ff4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.633+0000 7ff4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.633+0000 7ff4ff7fe640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.633+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4e8003520 con 0x7ff5001457f0 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044360515 0 0) 0x7ff5000aaef0 con 0x7ff50013f650 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:42:10.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff500145bd0 con 0x7ff50013f650 2024-03-29T11:42:10.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623183504 0 0) 0x7ff500145eb0 con 0x7ff5000a9d70 2024-03-29T11:42:10.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:42:10.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.634+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5000aaef0 con 0x7ff5000a9d70 2024-03-29T11:42:10.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 665770439 0 0) 0x7ff4e8003520 con 0x7ff5001457f0 2024-03-29T11:42:10.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 >> v1:172.21.15.155:6790/0 conn(0x7ff5000a9d70 legacy=0x7ff50014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 >> v1:172.21.15.155:6789/0 conn(0x7ff50013f650 legacy=0x7ff5000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:10.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 1 monclient: found mon.noname-c 2024-03-29T11:42:10.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:10.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:42:10.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff500146340 con 0x7ff5001457f0 2024-03-29T11:42:10.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:10.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:10.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:10.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff4f0001690 con 0x7ff5001457f0 2024-03-29T11:42:10.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:10.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.636+0000 7ff4ff7fe640 10 monclient: dump: 2024-03-29T11:42:10.640 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:10.640 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:10.640 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.640 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.640 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.641 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:10.641 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:10.641 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:10.641 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:10.641 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:10.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.637+0000 7ff5071e8640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:42:10.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.637+0000 7ff5071e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:10.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.640+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4f00017f0 con 0x7ff5001457f0 2024-03-29T11:42:10.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.640+0000 7ff4ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:10.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.640+0000 7ff4ff7fe640 1 -- 172.21.15.155:0/3847616724 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff4f0001c00 con 0x7ff5001457f0 2024-03-29T11:42:10.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.640+0000 7ff4ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.640+0000 7ff4ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:10.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.640+0000 7ff4ff7fe640 10 monclient: dump: 2024-03-29T11:42:10.646 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.641+0000 7ff5071e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.641+0000 7ff5071e8640 4 set_mon_vals no callback set 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.643+0000 7ff5071e8640 10 monclient: shutdown 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.643+0000 7ff5071e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:10.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.643+0000 7ff5071e8640 1 -- 172.21.15.155:0/3847616724 >> v1:172.21.15.183:6789/0 conn(0x7ff5001457f0 legacy=0x7ff50013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.643+0000 7ff5071e8640 1 -- 172.21.15.155:0/3847616724 shutdown_connections 2024-03-29T11:42:10.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.643+0000 7ff5071e8640 1 -- 172.21.15.155:0/3847616724 >> 172.21.15.155:0/3847616724 conn(0x7ff500066a00 msgr2=0x7ff500066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:10.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.645+0000 7ff5071e8640 1 -- 172.21.15.155:0/3847616724 shutdown_connections 2024-03-29T11:42:10.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.646+0000 7ff5071e8640 1 -- 172.21.15.155:0/3847616724 wait complete. 2024-03-29T11:42:10.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.646+0000 7ff5071e8640 10 monclient: build_initial_monmap 2024-03-29T11:42:10.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.646+0000 7ff5071e8640 10 monclient: monmap: 2024-03-29T11:42:10.650 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:10.650 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:10.650 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:42:10.650 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:42:10.650 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:10.650 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:10.650 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:10.650 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:42:10.650 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:42:10.650 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:10.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.647+0000 7ff5071e8640 1 librados: starting msgr at 2024-03-29T11:42:10.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.647+0000 7ff5071e8640 1 librados: starting objecter 2024-03-29T11:42:10.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.647+0000 7ff5071e8640 1 Processor -- start 2024-03-29T11:42:10.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.647+0000 7ff5071e8640 1 -- start start 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.647+0000 7ff5071e8640 1 librados: setting wanted keys 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.647+0000 7ff5071e8640 1 librados: calling monclient init 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.647+0000 7ff5071e8640 10 monclient: init 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff5071e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff5071e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff5071e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff5000a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff4fffff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ff50013f650 0x7ff500134fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:42018/0 (socket says 172.21.15.155:42018) 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff4fffff640 1 -- 172.21.15.155:0/2430481151 learned_addr learned my addr 172.21.15.155:0/2430481151 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff5071e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff50013f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff5071e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff5001457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50013b1b0 con 0x7ff5000a9d70 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50013b9a0 con 0x7ff5001457f0 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff500135bf0 con 0x7ff50013f650 2024-03-29T11:42:10.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff5071e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:10.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff5071e8640 10 monclient(hunting): authenticate will time out at 1312.775661s 2024-03-29T11:42:10.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff4fd7fa640 10 client.?.objecter ms_handle_connect 0x7ff50013f650 2024-03-29T11:42:10.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.648+0000 7ff4fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 client.?.objecter ms_handle_connect 0x7ff5000a9d70 2024-03-29T11:42:10.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 client.?.objecter ms_handle_connect 0x7ff5001457f0 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675478344 0 0) 0x7ff500135bf0 con 0x7ff50013f650 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ec004eb0 con 0x7ff50013f650 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400412087 0 0) 0x7ff50013b1b0 con 0x7ff5000a9d70 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff500135bf0 con 0x7ff5000a9d70 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455514890 0 0) 0x7ff50013b9a0 con 0x7ff5001457f0 2024-03-29T11:42:10.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.649+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff50013b1b0 con 0x7ff5001457f0 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1170800235 0 0) 0x7ff50013b1b0 con 0x7ff5001457f0 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4ec005620 con 0x7ff5001457f0 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff4f8001c70 con 0x7ff5001457f0 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 10 monclient(hunting): dump: 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.657 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1647582553 0 0) 0x7ff500135bf0 con 0x7ff5000a9d70 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff50013b1b0 con 0x7ff5000a9d70 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff4f0001b00 con 0x7ff5000a9d70 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.650+0000 7ff4fd7fa640 10 monclient(hunting): dump: 2024-03-29T11:42:10.658 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3258490732 0 0) 0x7ff4ec004eb0 con 0x7ff50013f650 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff500135bf0 con 0x7ff50013f650 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff4f4001970 con 0x7ff50013f650 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 10 monclient(hunting): dump: 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1318162267 0 0) 0x7ff4ec005620 con 0x7ff5001457f0 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 >> v1:172.21.15.155:6790/0 conn(0x7ff50013f650 legacy=0x7ff500134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 >> v1:172.21.15.155:6789/0 conn(0x7ff5000a9d70 legacy=0x7ff500139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 1 monclient: found mon.b 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff500138cc0 con 0x7ff5001457f0 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff4fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff5071e8640 5 monclient: authenticate success, global_id 4462 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff5071e8640 10 monclient: _renew_subs 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff5071e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50013a170 con 0x7ff5001457f0 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff5071e8640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff5071e8640 10 monclient: _renew_subs 2024-03-29T11:42:10.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff5071e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5001358c0 con 0x7ff5001457f0 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.651+0000 7ff5071e8640 1 librados: init done 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.652+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4f8002080 con 0x7ff5001457f0 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.652+0000 7ff4fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.652+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7ff4f8003b70 con 0x7ff5001457f0 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.652+0000 7ff4fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.652+0000 7ff4fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.652+0000 7ff4fd7fa640 10 monclient: dump: 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:10.662 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.663 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:10.663 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:10.663 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:10.663 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:10.663 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:10.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.652+0000 7ff4feffd640 4 set_mon_vals no callback set 2024-03-29T11:42:10.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.653+0000 7ff5071e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.653+0000 7ff5071e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.653+0000 7ff5071e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:10.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.653+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e0005080 con 0x7ff5001457f0 2024-03-29T11:42:10.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.654+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7ff4f8003cd0 con 0x7ff5001457f0 2024-03-29T11:42:10.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.655+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7ff4f8083a70 con 0x7ff5001457f0 2024-03-29T11:42:10.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.655+0000 7ff4fd7fa640 10 client.4462.objecter ms_dispatch 0x7ff500000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:10.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.655+0000 7ff4fd7fa640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:10.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.655+0000 7ff4fd7fa640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:10.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.655+0000 7ff4fd7fa640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-29T11:42:10.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.656+0000 7ff504f84640 10 monclient: get_auth_request con 0x7ff4ec06e230 auth_method 0 2024-03-29T11:42:10.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.656+0000 7ff4fd7fa640 10 client.4462.objecter ms_handle_connect 0x7ff4ec06e230 2024-03-29T11:42:10.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.662+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4f804f180 con 0x7ff5001457f0 2024-03-29T11:42:10.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.662+0000 7ff4fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:10.662+0000 7ff4fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:11.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.029+0000 7ff5071e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:11.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.029+0000 7ff5071e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:11.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.029+0000 7ff5071e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:11.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.029+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4e0005330 con 0x7ff5001457f0 2024-03-29T11:42:11.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.030+0000 7ff4fd7fa640 1 -- 172.21.15.155:0/2430481151 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7ff4f804f180 con 0x7ff5001457f0 2024-03-29T11:42:11.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.030+0000 7ff4fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:11.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.030+0000 7ff4fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:11.033 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:42:11.033 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:11.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.034+0000 7ff5071e8640 10 librados: watch_flush enter 2024-03-29T11:42:11.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.034+0000 7ff5071e8640 10 librados: watch_flush exit 2024-03-29T11:42:11.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.034+0000 7ff5071e8640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-29T11:42:11.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.034+0000 7ff5071e8640 10 client.4462.objecter successfully canceled tick 2024-03-29T11:42:11.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.034+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 >> v1:172.21.15.155:6816/1760814708 conn(0x7ff4ec06e230 legacy=0x7ff4ec0706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.034+0000 7ff5071e8640 10 monclient: shutdown 2024-03-29T11:42:11.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.034+0000 7ff5071e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:11.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.034+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 >> v1:172.21.15.183:6789/0 conn(0x7ff5001457f0 legacy=0x7ff500140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.034+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 shutdown_connections 2024-03-29T11:42:11.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.034+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 >> 172.21.15.155:0/2430481151 conn(0x7ff500066a00 msgr2=0x7ff5000b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:11.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.035+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 shutdown_connections 2024-03-29T11:42:11.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.035+0000 7ff5071e8640 1 -- 172.21.15.155:0/2430481151 wait complete. 2024-03-29T11:42:11.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:11.035+0000 7ff5071e8640 1 librados: shutdown 2024-03-29T11:42:11.056 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:17.866 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:18.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.010+0000 7f13c4393640 10 monclient: get_monmap_and_config 2024-03-29T11:42:18.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.010+0000 7f13c4393640 10 monclient: build_initial_monmap 2024-03-29T11:42:18.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.010+0000 7f13c4393640 10 monclient: monmap: 2024-03-29T11:42:18.014 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:18.014 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:18.014 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:42:18.011966+0000 2024-03-29T11:42:18.014 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:42:18.011966+0000 2024-03-29T11:42:18.014 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:18.014 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:18.014 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:18.014 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:42:18.015 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:42:18.015 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:18.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.011+0000 7f13c4393640 1 Processor -- start 2024-03-29T11:42:18.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.011+0000 7f13c4393640 1 -- start start 2024-03-29T11:42:18.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.011+0000 7f13c4393640 10 monclient: init 2024-03-29T11:42:18.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.012+0000 7f13c4393640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:18.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.012+0000 7f13c4393640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:18.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.013+0000 7f13c4393640 10 monclient(hunting): picked mon.noname-a con 0x7f13bc1422b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:18.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.013+0000 7f13c4393640 10 monclient(hunting): picked mon.noname-b con 0x7f13bc142690 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:18.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.013+0000 7f13c4393640 10 monclient(hunting): picked mon.noname-c con 0x7f13bc0b17d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:18.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.013+0000 7f13c4393640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc142c80 con 0x7f13bc1422b0 2024-03-29T11:42:18.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.013+0000 7f13c4393640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0aae40 con 0x7f13bc0b17d0 2024-03-29T11:42:18.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.013+0000 7f13c4393640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc13fbd0 con 0x7f13bc142690 2024-03-29T11:42:18.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.013+0000 7f13c4393640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:18.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.013+0000 7f13c4393640 10 monclient(hunting): authenticate will time out at 1320.140296s 2024-03-29T11:42:18.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.014+0000 7f13c212f640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f13bc1422b0 0x7f13bc13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:33968/0 (socket says 172.21.15.155:33968) 2024-03-29T11:42:18.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.014+0000 7f13c212f640 1 -- 172.21.15.155:0/2296864959 learned_addr learned my addr 172.21.15.155:0/2296864959 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:18.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.015+0000 7f13c112d640 1 -- 172.21.15.155:0/2296864959 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433420241 0 0) 0x7f13bc142c80 con 0x7f13bc1422b0 2024-03-29T11:42:18.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.015+0000 7f13c112d640 10 monclient(hunting): _negotiate 2024-03-29T11:42:18.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.015+0000 7f13c112d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:18.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.015+0000 7f13c112d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:18.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.015+0000 7f13c112d640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:42:18.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.015+0000 7f13c112d640 1 -- 172.21.15.155:0/2296864959 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a8003520 con 0x7f13bc1422b0 2024-03-29T11:42:18.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.016+0000 7f13c112d640 1 -- 172.21.15.155:0/2296864959 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3974710846 0 0) 0x7f13a8003520 con 0x7f13bc1422b0 2024-03-29T11:42:18.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.016+0000 7f13c112d640 1 -- 172.21.15.155:0/2296864959 >> v1:172.21.15.155:6790/0 conn(0x7f13bc142690 legacy=0x7f13bc0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:18.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.017+0000 7f13c112d640 1 -- 172.21.15.155:0/2296864959 >> v1:172.21.15.183:6789/0 conn(0x7f13bc0b17d0 legacy=0x7f13bc0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:18.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 10 monclient: _finish_hunting 0 2024-03-29T11:42:18.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 1 monclient: found mon.noname-a 2024-03-29T11:42:18.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:18.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:42:18.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 1 -- 172.21.15.155:0/2296864959 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc151940 con 0x7f13bc1422b0 2024-03-29T11:42:18.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 10 monclient: _finish_auth 0 2024-03-29T11:42:18.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 10 monclient: _check_auth_tickets 2024-03-29T11:42:18.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:18.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 1 -- 172.21.15.155:0/2296864959 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f13ac001690 con 0x7f13bc1422b0 2024-03-29T11:42:18.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:18.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:18.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 10 monclient: dump: 2024-03-29T11:42:18.024 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:18.024 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:18.024 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:18.024 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:18.024 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:18.024 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c4393640 5 monclient: authenticate success, global_id 4491 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c4393640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 1 -- 172.21.15.155:0/2296864959 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13ac0018b0 con 0x7f13bc1422b0 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 1 -- 172.21.15.155:0/2296864959 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f13ac001c40 con 0x7f13bc1422b0 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c112d640 10 monclient: dump: 2024-03-29T11:42:18.025 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c4393640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.018+0000 7f13c4393640 4 set_mon_vals no callback set 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.019+0000 7f13c4393640 10 monclient: shutdown 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.019+0000 7f13c4393640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:18.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.019+0000 7f13c4393640 1 -- 172.21.15.155:0/2296864959 >> v1:172.21.15.155:6789/0 conn(0x7f13bc1422b0 legacy=0x7f13bc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:18.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.019+0000 7f13c4393640 1 -- 172.21.15.155:0/2296864959 shutdown_connections 2024-03-29T11:42:18.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.019+0000 7f13c4393640 1 -- 172.21.15.155:0/2296864959 >> 172.21.15.155:0/2296864959 conn(0x7f13bc066a00 msgr2=0x7f13bc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:18.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.022+0000 7f13c4393640 1 -- 172.21.15.155:0/2296864959 shutdown_connections 2024-03-29T11:42:18.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.022+0000 7f13c4393640 1 -- 172.21.15.155:0/2296864959 wait complete. 2024-03-29T11:42:18.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.023+0000 7f13c4393640 10 monclient: build_initial_monmap 2024-03-29T11:42:18.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.023+0000 7f13c4393640 10 monclient: monmap: 2024-03-29T11:42:18.028 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:18.028 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:18.028 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:42:18.028 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:42:18.028 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:18.028 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:18.029 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:18.029 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:42:18.029 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:42:18.029 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:18.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.024+0000 7f13c4393640 1 librados: starting msgr at 2024-03-29T11:42:18.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.024+0000 7f13c4393640 1 librados: starting objecter 2024-03-29T11:42:18.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.024+0000 7f13c4393640 1 Processor -- start 2024-03-29T11:42:18.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.024+0000 7f13c4393640 1 -- start start 2024-03-29T11:42:18.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.024+0000 7f13c4393640 1 librados: setting wanted keys 2024-03-29T11:42:18.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.024+0000 7f13c4393640 1 librados: calling monclient init 2024-03-29T11:42:18.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.024+0000 7f13c4393640 10 monclient: init 2024-03-29T11:42:18.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.025+0000 7f13c4393640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:18.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.025+0000 7f13c4393640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:18.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.026+0000 7f13c4393640 10 monclient(hunting): picked mon.noname-c con 0x7f13bc0b17d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:18.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.026+0000 7f13c4393640 10 monclient(hunting): picked mon.noname-b con 0x7f13bc1422b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:18.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.026+0000 7f13c4393640 10 monclient(hunting): picked mon.noname-a con 0x7f13bc142690 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:18.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.026+0000 7f13c4393640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0accd0 con 0x7f13bc142690 2024-03-29T11:42:18.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.026+0000 7f13c4393640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0acfb0 con 0x7f13bc1422b0 2024-03-29T11:42:18.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.026+0000 7f13c4393640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0abdc0 con 0x7f13bc0b17d0 2024-03-29T11:42:18.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.026+0000 7f13c4393640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.026+0000 7f13c4393640 10 monclient(hunting): authenticate will time out at 1320.153409s 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.028+0000 7f13c212f640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f13bc0b17d0 0x7f13bc1daad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:35907/0 (socket says 172.21.15.155:35907) 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.028+0000 7f13c212f640 1 -- 172.21.15.155:0/106508642 learned_addr learned my addr 172.21.15.155:0/106508642 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.029+0000 7f13b2ffd640 10 client.?.objecter ms_handle_connect 0x7f13bc0b17d0 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.029+0000 7f13b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.029+0000 7f13b2ffd640 10 client.?.objecter ms_handle_connect 0x7f13bc142690 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.029+0000 7f13b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.029+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742386205 0 0) 0x7f13bc0accd0 con 0x7f13bc142690 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.029+0000 7f13b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.029+0000 7f13b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.029+0000 7f13b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.029+0000 7f13b2ffd640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.029+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a0004eb0 con 0x7f13bc142690 2024-03-29T11:42:18.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.030+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715146981 0 0) 0x7f13bc0abdc0 con 0x7f13bc0b17d0 2024-03-29T11:42:18.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.030+0000 7f13b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:18.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.030+0000 7f13b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:18.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.030+0000 7f13b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:18.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.030+0000 7f13b2ffd640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:42:18.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.030+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc0accd0 con 0x7f13bc0b17d0 2024-03-29T11:42:18.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.030+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 910847681 0 0) 0x7f13bc0accd0 con 0x7f13bc0b17d0 2024-03-29T11:42:18.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.030+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13bc0abdc0 con 0x7f13bc0b17d0 2024-03-29T11:42:18.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.030+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f13ac001550 con 0x7f13bc0b17d0 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.030+0000 7f13b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:18.038 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1883684895 0 0) 0x7f13bc0abdc0 con 0x7f13bc0b17d0 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 >> v1:172.21.15.183:6789/0 conn(0x7f13bc1422b0 legacy=0x7f13bc1db1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 >> v1:172.21.15.155:6789/0 conn(0x7f13bc142690 legacy=0x7f13bc1e1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 1 monclient: found mon.c 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0aba00 con 0x7f13bc0b17d0 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:18.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13c4393640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13c4393640 10 monclient: _renew_subs 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13c4393640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13c4393640 1 -- 172.21.15.155:0/106508642 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc132970 con 0x7f13bc0b17d0 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13c4393640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13c4393640 10 monclient: _renew_subs 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13c4393640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13c4393640 1 -- 172.21.15.155:0/106508642 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc1325c0 con 0x7f13bc0b17d0 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13ac004bf0 con 0x7f13bc0b17d0 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13c4393640 1 librados: init done 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f13ac004f00 con 0x7f13bc0b17d0 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:18.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13b2ffd640 10 monclient: dump: 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.031+0000 7f13c092c640 4 set_mon_vals no callback set 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.034+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f13ac01aed0 con 0x7f13bc0b17d0 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.034+0000 7f13b0ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.034+0000 7f13b0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:18.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.034+0000 7f13b0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:18.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.034+0000 7f13b0ff9640 1 -- 172.21.15.155:0/106508642 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1398005080 con 0x7f13bc0b17d0 2024-03-29T11:42:18.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.035+0000 7f13c192e640 10 monclient: get_auth_request con 0x7f13a006dec0 auth_method 0 2024-03-29T11:42:18.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.035+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f13ac084c20 con 0x7f13bc0b17d0 2024-03-29T11:42:18.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.035+0000 7f13b2ffd640 10 client.4547.objecter ms_dispatch 0x7f13bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:18.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.035+0000 7f13b2ffd640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:18.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.035+0000 7f13b2ffd640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:18.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.035+0000 7f13b2ffd640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:42:18.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.036+0000 7f13b2ffd640 10 client.4547.objecter ms_handle_connect 0x7f13a006dec0 2024-03-29T11:42:18.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.041+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13ac050500 con 0x7f13bc0b17d0 2024-03-29T11:42:18.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.041+0000 7f13b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:18.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.041+0000 7f13b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:18.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.367+0000 7f13b0ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:18.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.367+0000 7f13b0ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:18.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.367+0000 7f13b0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:18.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.367+0000 7f13b0ff9640 1 -- 172.21.15.155:0/106508642 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1398005330 con 0x7f13bc0b17d0 2024-03-29T11:42:18.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.368+0000 7f13b2ffd640 1 -- 172.21.15.155:0/106508642 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f13ac052430 con 0x7f13bc0b17d0 2024-03-29T11:42:18.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.368+0000 7f13b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:18.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.368+0000 7f13b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:18.372 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:42:18.372 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:18.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.372+0000 7f13b0ff9640 10 librados: watch_flush enter 2024-03-29T11:42:18.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.372+0000 7f13b0ff9640 10 librados: watch_flush exit 2024-03-29T11:42:18.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.372+0000 7f13b0ff9640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:42:18.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.372+0000 7f13b0ff9640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:42:18.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.373+0000 7f13b0ff9640 1 -- 172.21.15.155:0/106508642 >> v1:172.21.15.155:6816/1760814708 conn(0x7f13a006dec0 legacy=0x7f13a0070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:18.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.373+0000 7f13b0ff9640 10 monclient: shutdown 2024-03-29T11:42:18.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.373+0000 7f13b0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:18.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.373+0000 7f13b0ff9640 1 -- 172.21.15.155:0/106508642 >> v1:172.21.15.155:6790/0 conn(0x7f13bc0b17d0 legacy=0x7f13bc1daad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:18.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.373+0000 7f13b0ff9640 1 -- 172.21.15.155:0/106508642 shutdown_connections 2024-03-29T11:42:18.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.373+0000 7f13b0ff9640 1 -- 172.21.15.155:0/106508642 >> 172.21.15.155:0/106508642 conn(0x7f13bc066a00 msgr2=0x7f13bc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:18.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.373+0000 7f13b0ff9640 1 -- 172.21.15.155:0/106508642 shutdown_connections 2024-03-29T11:42:18.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.373+0000 7f13b0ff9640 1 -- 172.21.15.155:0/106508642 wait complete. 2024-03-29T11:42:18.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:18.373+0000 7f13b0ff9640 1 librados: shutdown 2024-03-29T11:42:18.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:25.201 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:25.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.346+0000 7f5ec1017640 10 monclient: get_monmap_and_config 2024-03-29T11:42:25.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.346+0000 7f5ec1017640 10 monclient: build_initial_monmap 2024-03-29T11:42:25.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.346+0000 7f5ec1017640 10 monclient: monmap: 2024-03-29T11:42:25.348 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:25.348 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:25.349 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:42:25.346545+0000 2024-03-29T11:42:25.349 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:42:25.346545+0000 2024-03-29T11:42:25.349 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:25.349 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:25.349 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:25.349 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:42:25.349 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:42:25.349 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:25.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.349+0000 7f5ec1017640 1 Processor -- start 2024-03-29T11:42:25.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.351+0000 7f5ec1017640 1 -- start start 2024-03-29T11:42:25.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.351+0000 7f5ec1017640 10 monclient: init 2024-03-29T11:42:25.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.352+0000 7f5ec1017640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:25.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.352+0000 7f5ec1017640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:25.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.352+0000 7f5ec1017640 10 monclient(hunting): picked mon.noname-a con 0x7f5ebc1422b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:25.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.352+0000 7f5ec1017640 10 monclient(hunting): picked mon.noname-b con 0x7f5ebc142690 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:25.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.352+0000 7f5ec1017640 10 monclient(hunting): picked mon.noname-c con 0x7f5ebc0b17d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:25.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.352+0000 7f5ec1017640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc142c80 con 0x7f5ebc1422b0 2024-03-29T11:42:25.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.352+0000 7f5ec1017640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc0aae40 con 0x7f5ebc0b17d0 2024-03-29T11:42:25.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.352+0000 7f5ec1017640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc13fbd0 con 0x7f5ebc142690 2024-03-29T11:42:25.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.352+0000 7f5ec1017640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:25.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.352+0000 7f5ec1017640 10 monclient(hunting): authenticate will time out at 1327.478353s 2024-03-29T11:42:25.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.356+0000 7f5eb259c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f5ebc142690 0x7f5ebc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:58597/0 (socket says 172.21.15.155:58597) 2024-03-29T11:42:25.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.356+0000 7f5eb259c640 1 -- 172.21.15.155:0/2576891723 learned_addr learned my addr 172.21.15.155:0/2576891723 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:25.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 1 -- 172.21.15.155:0/2576891723 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125305133 0 0) 0x7f5ebc13fbd0 con 0x7f5ebc142690 2024-03-29T11:42:25.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 10 monclient(hunting): _negotiate 2024-03-29T11:42:25.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:25.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:25.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:42:25.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 1 -- 172.21.15.155:0/2576891723 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea8003520 con 0x7f5ebc142690 2024-03-29T11:42:25.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 1 -- 172.21.15.155:0/2576891723 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3468533667 0 0) 0x7f5ea8003520 con 0x7f5ebc142690 2024-03-29T11:42:25.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 1 -- 172.21.15.155:0/2576891723 >> v1:172.21.15.183:6789/0 conn(0x7f5ebc0b17d0 legacy=0x7f5ebc0b2ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:42:25.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 1 -- 172.21.15.155:0/2576891723 >> v1:172.21.15.155:6789/0 conn(0x7f5ebc1422b0 legacy=0x7f5ebc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:25.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 10 monclient: _finish_hunting 0 2024-03-29T11:42:25.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 1 monclient: found mon.noname-b 2024-03-29T11:42:25.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:25.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:42:25.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 1 -- 172.21.15.155:0/2576891723 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc151940 con 0x7f5ebc142690 2024-03-29T11:42:25.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 10 monclient: _finish_auth 0 2024-03-29T11:42:25.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 10 monclient: _check_auth_tickets 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 1 -- 172.21.15.155:0/2576891723 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f5eac001690 con 0x7f5ebc142690 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.357+0000 7f5eba59c640 10 monclient: dump: 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:25.364 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.358+0000 7f5ec1017640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.358+0000 7f5ec1017640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.358+0000 7f5eba59c640 1 -- 172.21.15.155:0/2576891723 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eac0017f0 con 0x7f5ebc142690 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.358+0000 7f5eba59c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.358+0000 7f5eba59c640 1 -- 172.21.15.155:0/2576891723 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f5eac001c40 con 0x7f5ebc142690 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.358+0000 7f5eba59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.358+0000 7f5eba59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.358+0000 7f5eba59c640 10 monclient: dump: 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:25.365 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:25.366 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:25.366 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:25.366 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:25.366 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:25.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.360+0000 7f5ec1017640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:25.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.360+0000 7f5ec1017640 4 set_mon_vals no callback set 2024-03-29T11:42:25.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.361+0000 7f5ec1017640 10 monclient: shutdown 2024-03-29T11:42:25.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.361+0000 7f5ec1017640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:25.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.361+0000 7f5ec1017640 1 -- 172.21.15.155:0/2576891723 >> v1:172.21.15.155:6790/0 conn(0x7f5ebc142690 legacy=0x7f5ebc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:25.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.361+0000 7f5ec1017640 1 -- 172.21.15.155:0/2576891723 shutdown_connections 2024-03-29T11:42:25.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.361+0000 7f5ec1017640 1 -- 172.21.15.155:0/2576891723 >> 172.21.15.155:0/2576891723 conn(0x7f5ebc066a00 msgr2=0x7f5ebc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:25.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.362+0000 7f5ec1017640 1 -- 172.21.15.155:0/2576891723 shutdown_connections 2024-03-29T11:42:25.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.362+0000 7f5ec1017640 1 -- 172.21.15.155:0/2576891723 wait complete. 2024-03-29T11:42:25.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.363+0000 7f5ec1017640 10 monclient: build_initial_monmap 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.363+0000 7f5ec1017640 10 monclient: monmap: 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.363+0000 7f5ec1017640 1 librados: starting msgr at 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.363+0000 7f5ec1017640 1 librados: starting objecter 2024-03-29T11:42:25.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.363+0000 7f5ec1017640 1 Processor -- start 2024-03-29T11:42:25.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.364+0000 7f5ec1017640 1 -- start start 2024-03-29T11:42:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.364+0000 7f5ec1017640 1 librados: setting wanted keys 2024-03-29T11:42:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.364+0000 7f5ec1017640 1 librados: calling monclient init 2024-03-29T11:42:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.364+0000 7f5ec1017640 10 monclient: init 2024-03-29T11:42:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.365+0000 7f5ec1017640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.365+0000 7f5ec1017640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.365+0000 7f5ec1017640 10 monclient(hunting): picked mon.noname-a con 0x7f5ebc0b17d0 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.365+0000 7f5ec1017640 10 monclient(hunting): picked mon.noname-b con 0x7f5ebc1422b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.365+0000 7f5ec1017640 10 monclient(hunting): picked mon.noname-c con 0x7f5ebc142690 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.365+0000 7f5ec1017640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc151470 con 0x7f5ebc0b17d0 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.365+0000 7f5ec1017640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc06eba0 con 0x7f5ebc1422b0 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.365+0000 7f5ec1017640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc06ee80 con 0x7f5ebc142690 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.365+0000 7f5ec1017640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.365+0000 7f5ec1017640 10 monclient(hunting): authenticate will time out at 1327.491207s 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.369+0000 7f5ebb59e640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f5ebc142690 0x7f5ebc1e6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:58687/0 (socket says 172.21.15.155:58687) 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.369+0000 7f5ebb59e640 1 -- 172.21.15.155:0/238063391 learned_addr learned my addr 172.21.15.155:0/238063391 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.369+0000 7f5eb3fff640 10 client.?.objecter ms_handle_connect 0x7f5ebc142690 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.369+0000 7f5eb3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.369+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057827163 0 0) 0x7f5ebc06ee80 con 0x7f5ebc142690 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.369+0000 7f5eb3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:25.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.369+0000 7f5eb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:25.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.371+0000 7f5eb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:25.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.371+0000 7f5eb3fff640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:42:25.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.371+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea8003520 con 0x7f5ebc142690 2024-03-29T11:42:25.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2294021956 0 0) 0x7f5ea8003520 con 0x7f5ebc142690 2024-03-29T11:42:25.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ebc06ee80 con 0x7f5ebc142690 2024-03-29T11:42:25.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f5eb4001b70 con 0x7f5ebc142690 2024-03-29T11:42:25.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:25.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient(hunting): dump: 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 client.?.objecter ms_handle_connect 0x7f5ebc0b17d0 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:25.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282428506 0 0) 0x7f5ebc151470 con 0x7f5ebc0b17d0 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea8003520 con 0x7f5ebc0b17d0 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 785869981 0 0) 0x7f5ebc06ee80 con 0x7f5ebc142690 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 >> v1:172.21.15.183:6789/0 conn(0x7f5ebc1422b0 legacy=0x7f5ebc12ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 >> v1:172.21.15.155:6789/0 conn(0x7f5ebc0b17d0 legacy=0x7f5ebc140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 monclient: found mon.c 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc150f50 con 0x7f5ebc142690 2024-03-29T11:42:25.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5ec1017640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5ec1017640 10 monclient: _renew_subs 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5ec1017640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5ec1017640 1 -- 172.21.15.155:0/238063391 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ebc150b70 con 0x7f5ebc142690 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5ec1017640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5ec1017640 10 monclient: _renew_subs 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5ec1017640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5ec1017640 1 -- 172.21.15.155:0/238063391 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ebc06e070 con 0x7f5ebc142690 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5ec1017640 1 librados: init done 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eb4001cd0 con 0x7f5ebc142690 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f5eb4004e20 con 0x7f5ebc142690 2024-03-29T11:42:25.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.372+0000 7f5eb3fff640 10 monclient: dump: 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.374+0000 7f5eb9d9b640 4 set_mon_vals no callback set 2024-03-29T11:42:25.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.376+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f5eb4004f80 con 0x7f5ebc142690 2024-03-29T11:42:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.376+0000 7f5ec1017640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.376+0000 7f5ec1017640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.376+0000 7f5ec1017640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.377+0000 7f5ec1017640 1 -- 172.21.15.155:0/238063391 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea0005080 con 0x7f5ebc142690 2024-03-29T11:42:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.377+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f5eb4084cc0 con 0x7f5ebc142690 2024-03-29T11:42:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.377+0000 7f5ebad9d640 10 monclient: get_auth_request con 0x7f5ea806de50 auth_method 0 2024-03-29T11:42:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.379+0000 7f5eb3fff640 10 client.4553.objecter ms_dispatch 0x7f5ebc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.379+0000 7f5eb3fff640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.379+0000 7f5eb3fff640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.379+0000 7f5eb3fff640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-29T11:42:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.379+0000 7f5eb3fff640 10 client.4553.objecter ms_handle_connect 0x7f5ea806de50 2024-03-29T11:42:25.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.386+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5eb40505a0 con 0x7f5ebc142690 2024-03-29T11:42:25.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.386+0000 7f5eb3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:25.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.386+0000 7f5eb3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:25.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.719+0000 7f5ec1017640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:25.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.719+0000 7f5ec1017640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:25.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.719+0000 7f5ec1017640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:25.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.719+0000 7f5ec1017640 1 -- 172.21.15.155:0/238063391 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ea0005330 con 0x7f5ebc142690 2024-03-29T11:42:25.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.722+0000 7f5eb3fff640 1 -- 172.21.15.155:0/238063391 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f5eb40524d0 con 0x7f5ebc142690 2024-03-29T11:42:25.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.722+0000 7f5eb3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:25.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.722+0000 7f5eb3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:25.725 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:42:25.725 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:25.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.726+0000 7f5ec1017640 10 librados: watch_flush enter 2024-03-29T11:42:25.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.726+0000 7f5ec1017640 10 librados: watch_flush exit 2024-03-29T11:42:25.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.726+0000 7f5ec1017640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-29T11:42:25.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.726+0000 7f5ec1017640 10 client.4553.objecter successfully canceled tick 2024-03-29T11:42:25.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.726+0000 7f5ec1017640 1 -- 172.21.15.155:0/238063391 >> v1:172.21.15.155:6816/1760814708 conn(0x7f5ea806de50 legacy=0x7f5ea8070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:25.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.726+0000 7f5ec1017640 10 monclient: shutdown 2024-03-29T11:42:25.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.726+0000 7f5ec1017640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:25.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.726+0000 7f5ec1017640 1 -- 172.21.15.155:0/238063391 >> v1:172.21.15.155:6790/0 conn(0x7f5ebc142690 legacy=0x7f5ebc1e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:25.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.727+0000 7f5ec1017640 1 -- 172.21.15.155:0/238063391 shutdown_connections 2024-03-29T11:42:25.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.727+0000 7f5ec1017640 1 -- 172.21.15.155:0/238063391 >> 172.21.15.155:0/238063391 conn(0x7f5ebc066a00 msgr2=0x7f5ebc0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:25.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.729+0000 7f5ec1017640 1 -- 172.21.15.155:0/238063391 shutdown_connections 2024-03-29T11:42:25.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.730+0000 7f5ec1017640 1 -- 172.21.15.155:0/238063391 wait complete. 2024-03-29T11:42:25.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:25.730+0000 7f5ec1017640 1 librados: shutdown 2024-03-29T11:42:25.752 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:32.559 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:32.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.702+0000 7f3968206640 10 monclient: get_monmap_and_config 2024-03-29T11:42:32.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.702+0000 7f3968206640 10 monclient: build_initial_monmap 2024-03-29T11:42:32.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.702+0000 7f3968206640 10 monclient: monmap: 2024-03-29T11:42:32.705 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:32.705 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:32.705 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:42:32.702665+0000 2024-03-29T11:42:32.705 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:42:32.702665+0000 2024-03-29T11:42:32.705 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:32.705 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:32.705 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:32.705 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:42:32.705 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:42:32.705 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:32.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.703+0000 7f3968206640 1 Processor -- start 2024-03-29T11:42:32.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.703+0000 7f3968206640 1 -- start start 2024-03-29T11:42:32.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.703+0000 7f3968206640 10 monclient: init 2024-03-29T11:42:32.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f3968206640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:32.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f3968206640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:32.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f3968206640 10 monclient(hunting): picked mon.noname-c con 0x7f39601457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:32.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f3968206640 10 monclient(hunting): picked mon.noname-b con 0x7f396013f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:32.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f3968206640 10 monclient(hunting): picked mon.noname-a con 0x7f39600a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:32.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f3968206640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39600aaef0 con 0x7f39600a9d70 2024-03-29T11:42:32.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f3968206640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3960145bd0 con 0x7f39601457f0 2024-03-29T11:42:32.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f3968206640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3960145eb0 con 0x7f396013f650 2024-03-29T11:42:32.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f3968206640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:32.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f3968206640 10 monclient(hunting): authenticate will time out at 1334.830160s 2024-03-29T11:42:32.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f39657a1640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f396013f650 0x7f39600b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:37392/0 (socket says 172.21.15.155:37392) 2024-03-29T11:42:32.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f39657a1640 1 -- 172.21.15.155:0/1980811353 learned_addr learned my addr 172.21.15.155:0/1980811353 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:32.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.704+0000 7f3964fa0640 1 -- 172.21.15.155:0/1980811353 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843675882 0 0) 0x7f3960145eb0 con 0x7f396013f650 2024-03-29T11:42:32.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.705+0000 7f3964fa0640 10 monclient(hunting): _negotiate 2024-03-29T11:42:32.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.705+0000 7f3964fa0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:32.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.705+0000 7f3964fa0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:32.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.705+0000 7f3964fa0640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:42:32.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.705+0000 7f3964fa0640 1 -- 172.21.15.155:0/1980811353 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394c003520 con 0x7f396013f650 2024-03-29T11:42:32.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.705+0000 7f3964fa0640 1 -- 172.21.15.155:0/1980811353 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2595355091 0 0) 0x7f394c003520 con 0x7f396013f650 2024-03-29T11:42:32.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.705+0000 7f3964fa0640 1 -- 172.21.15.155:0/1980811353 >> v1:172.21.15.183:6789/0 conn(0x7f39601457f0 legacy=0x7f396013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:32.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.707+0000 7f3964fa0640 1 -- 172.21.15.155:0/1980811353 >> v1:172.21.15.155:6789/0 conn(0x7f39600a9d70 legacy=0x7f396014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:32.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 10 monclient: _finish_hunting 0 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 1 monclient: found mon.noname-b 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 1 -- 172.21.15.155:0/1980811353 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3960146340 con 0x7f396013f650 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 10 monclient: _finish_auth 0 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 10 monclient: _check_auth_tickets 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 1 -- 172.21.15.155:0/1980811353 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f395c001690 con 0x7f396013f650 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3964fa0640 10 monclient: dump: 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:32.713 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:32.714 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:32.714 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:32.714 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:32.714 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:32.714 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:32.714 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:32.714 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:32.714 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:32.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3968206640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:42:32.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.710+0000 7f3968206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:32.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.713+0000 7f3964fa0640 1 -- 172.21.15.155:0/1980811353 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f395c001860 con 0x7f396013f650 2024-03-29T11:42:32.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.713+0000 7f3964fa0640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:32.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.713+0000 7f3968206640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:32.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.713+0000 7f3968206640 4 set_mon_vals no callback set 2024-03-29T11:42:32.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.713+0000 7f3964fa0640 1 -- 172.21.15.155:0/1980811353 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f395c001c70 con 0x7f396013f650 2024-03-29T11:42:32.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.713+0000 7f3964fa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:32.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.713+0000 7f3964fa0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:32.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.713+0000 7f3964fa0640 10 monclient: dump: 2024-03-29T11:42:32.717 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:32.717 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.714+0000 7f3968206640 10 monclient: shutdown 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.714+0000 7f3968206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.714+0000 7f3968206640 1 -- 172.21.15.155:0/1980811353 >> v1:172.21.15.155:6790/0 conn(0x7f396013f650 legacy=0x7f39600b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:32.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.714+0000 7f3968206640 1 -- 172.21.15.155:0/1980811353 shutdown_connections 2024-03-29T11:42:32.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.714+0000 7f3968206640 1 -- 172.21.15.155:0/1980811353 >> 172.21.15.155:0/1980811353 conn(0x7f3960066a00 msgr2=0x7f3960066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:32.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.714+0000 7f3968206640 1 -- 172.21.15.155:0/1980811353 shutdown_connections 2024-03-29T11:42:32.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.714+0000 7f3968206640 1 -- 172.21.15.155:0/1980811353 wait complete. 2024-03-29T11:42:32.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.715+0000 7f3968206640 10 monclient: build_initial_monmap 2024-03-29T11:42:32.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.715+0000 7f3968206640 10 monclient: monmap: 2024-03-29T11:42:32.721 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:32.721 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:32.721 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:42:32.721 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:42:32.721 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:32.721 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:32.722 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:32.722 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:42:32.722 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:42:32.722 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:32.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.716+0000 7f3968206640 1 librados: starting msgr at 2024-03-29T11:42:32.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.716+0000 7f3968206640 1 librados: starting objecter 2024-03-29T11:42:32.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.716+0000 7f3968206640 1 Processor -- start 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.716+0000 7f3968206640 1 -- start start 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.716+0000 7f3968206640 1 librados: setting wanted keys 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.716+0000 7f3968206640 1 librados: calling monclient init 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.716+0000 7f3968206640 10 monclient: init 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.717+0000 7f3968206640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.717+0000 7f3968206640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.717+0000 7f3968206640 10 monclient(hunting): picked mon.noname-c con 0x7f39600a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.717+0000 7f3968206640 10 monclient(hunting): picked mon.noname-b con 0x7f396013f650 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.717+0000 7f3968206640 10 monclient(hunting): picked mon.noname-a con 0x7f39601457f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.717+0000 7f3968206640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396006f9b0 con 0x7f39601457f0 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.717+0000 7f3968206640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3960141860 con 0x7f396013f650 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.717+0000 7f3968206640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396006fc00 con 0x7f39600a9d70 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.717+0000 7f3968206640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:32.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.717+0000 7f3968206640 10 monclient(hunting): authenticate will time out at 1334.843167s 2024-03-29T11:42:32.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.720+0000 7f3965fa2640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f39600a9d70 0x7f396006e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39322/0 (socket says 172.21.15.155:39322) 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.720+0000 7f3965fa2640 1 -- 172.21.15.155:0/1316223446 learned_addr learned my addr 172.21.15.155:0/1316223446 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.720+0000 7f39527fc640 10 client.?.objecter ms_handle_connect 0x7f39600a9d70 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.720+0000 7f39527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 client.?.objecter ms_handle_connect 0x7f39601457f0 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261014542 0 0) 0x7f396006f9b0 con 0x7f39601457f0 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3944004eb0 con 0x7f39601457f0 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 client.?.objecter ms_handle_connect 0x7f396013f650 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:32.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370256492 0 0) 0x7f3960141860 con 0x7f396013f650 2024-03-29T11:42:32.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:32.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:32.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:32.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:42:32.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f396006f9b0 con 0x7f396013f650 2024-03-29T11:42:32.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.721+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3137870118 0 0) 0x7f3944004eb0 con 0x7f39601457f0 2024-03-29T11:42:32.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3944005490 con 0x7f39601457f0 2024-03-29T11:42:32.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f39540017e0 con 0x7f39601457f0 2024-03-29T11:42:32.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:32.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:32.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 10 monclient(hunting): dump: 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2727222289 0 0) 0x7f3944005490 con 0x7f39601457f0 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 >> v1:172.21.15.155:6790/0 conn(0x7f39600a9d70 legacy=0x7f396006e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:32.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 >> v1:172.21.15.183:6789/0 conn(0x7f396013f650 legacy=0x7f3960140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:32.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:32.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 1 monclient: found mon.a 2024-03-29T11:42:32.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:32.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:42:32.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3960070370 con 0x7f39601457f0 2024-03-29T11:42:32.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:32.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:32.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f39527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:32.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f3968206640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:42:32.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f3968206640 10 monclient: _renew_subs 2024-03-29T11:42:32.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f3968206640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:42:32.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f3968206640 1 -- 172.21.15.155:0/1316223446 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39600705c0 con 0x7f39601457f0 2024-03-29T11:42:32.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f3968206640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:32.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f3968206640 10 monclient: _renew_subs 2024-03-29T11:42:32.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f3968206640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:42:32.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f3968206640 1 -- 172.21.15.155:0/1316223446 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f396006ebd0 con 0x7f39601457f0 2024-03-29T11:42:32.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.722+0000 7f3968206640 1 librados: init done 2024-03-29T11:42:32.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.724+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3954001cd0 con 0x7f39601457f0 2024-03-29T11:42:32.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.724+0000 7f39527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:32.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.724+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3954003790 con 0x7f39601457f0 2024-03-29T11:42:32.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.724+0000 7f39527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:32.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.724+0000 7f39527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:32.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.724+0000 7f39527fc640 10 monclient: dump: 2024-03-29T11:42:32.733 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:32.733 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.724+0000 7f3953fff640 4 set_mon_vals no callback set 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.724+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f39540039b0 con 0x7f39601457f0 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.725+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f3954083740 con 0x7f39601457f0 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.725+0000 7f39527fc640 10 client.4500.objecter ms_dispatch 0x7f3960000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.725+0000 7f39527fc640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:32.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.725+0000 7f39527fc640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:32.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.725+0000 7f39527fc640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T11:42:32.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.725+0000 7f3965fa2640 10 monclient: get_auth_request con 0x7f394406dee0 auth_method 0 2024-03-29T11:42:32.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.726+0000 7f3968206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:32.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.727+0000 7f3968206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:32.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.727+0000 7f3968206640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:42:32.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.727+0000 7f3968206640 1 -- 172.21.15.155:0/1316223446 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3948005080 con 0x7f39601457f0 2024-03-29T11:42:32.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.728+0000 7f39527fc640 10 client.4500.objecter ms_handle_connect 0x7f394406dee0 2024-03-29T11:42:32.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.735+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f395404f1c0 con 0x7f39601457f0 2024-03-29T11:42:32.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.735+0000 7f39527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:32.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:32.735+0000 7f39527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:33.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.055+0000 7f3968206640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:33.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.055+0000 7f3968206640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:33.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.055+0000 7f3968206640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:42:33.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.055+0000 7f3968206640 1 -- 172.21.15.155:0/1316223446 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3948005330 con 0x7f39601457f0 2024-03-29T11:42:33.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.059+0000 7f39527fc640 1 -- 172.21.15.155:0/1316223446 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f39540510f0 con 0x7f39601457f0 2024-03-29T11:42:33.062 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:42:33.062 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:33.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.059+0000 7f39527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:33.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.059+0000 7f39527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:33.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.063+0000 7f3968206640 10 librados: watch_flush enter 2024-03-29T11:42:33.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.064+0000 7f3968206640 10 librados: watch_flush exit 2024-03-29T11:42:33.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.064+0000 7f3968206640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T11:42:33.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.064+0000 7f3968206640 10 client.4500.objecter successfully canceled tick 2024-03-29T11:42:33.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.064+0000 7f3968206640 1 -- 172.21.15.155:0/1316223446 >> v1:172.21.15.155:6816/1760814708 conn(0x7f394406dee0 legacy=0x7f39440703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:33.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.064+0000 7f3968206640 10 monclient: shutdown 2024-03-29T11:42:33.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.064+0000 7f3968206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:33.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.064+0000 7f3968206640 1 -- 172.21.15.155:0/1316223446 >> v1:172.21.15.155:6789/0 conn(0x7f39601457f0 legacy=0x7f39601ee9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:33.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.064+0000 7f3968206640 1 -- 172.21.15.155:0/1316223446 shutdown_connections 2024-03-29T11:42:33.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.064+0000 7f3968206640 1 -- 172.21.15.155:0/1316223446 >> 172.21.15.155:0/1316223446 conn(0x7f3960066a00 msgr2=0x7f396013e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:33.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.068+0000 7f3968206640 1 -- 172.21.15.155:0/1316223446 shutdown_connections 2024-03-29T11:42:33.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.068+0000 7f3968206640 1 -- 172.21.15.155:0/1316223446 wait complete. 2024-03-29T11:42:33.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:33.068+0000 7f3968206640 1 librados: shutdown 2024-03-29T11:42:33.090 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:39.897 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:40.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.042+0000 7f18cc8c0640 10 monclient: get_monmap_and_config 2024-03-29T11:42:40.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.042+0000 7f18cc8c0640 10 monclient: build_initial_monmap 2024-03-29T11:42:40.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.042+0000 7f18cc8c0640 10 monclient: monmap: 2024-03-29T11:42:40.045 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:40.046 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:40.046 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:42:40.042602+0000 2024-03-29T11:42:40.046 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:42:40.042602+0000 2024-03-29T11:42:40.046 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:40.046 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:40.046 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:40.046 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:42:40.046 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:42:40.046 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:40.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.043+0000 7f18cc8c0640 1 Processor -- start 2024-03-29T11:42:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.043+0000 7f18cc8c0640 1 -- start start 2024-03-29T11:42:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.043+0000 7f18cc8c0640 10 monclient: init 2024-03-29T11:42:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.044+0000 7f18cc8c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.044+0000 7f18cc8c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.044+0000 7f18cc8c0640 10 monclient(hunting): picked mon.noname-c con 0x7f18c80a9930 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.044+0000 7f18cc8c0640 10 monclient(hunting): picked mon.noname-b con 0x7f18c8149dc0 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.044+0000 7f18cc8c0640 10 monclient(hunting): picked mon.noname-a con 0x7f18c814ff40 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.044+0000 7f18cc8c0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c80aae60 con 0x7f18c814ff40 2024-03-29T11:42:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.044+0000 7f18cc8c0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c80b4310 con 0x7f18c80a9930 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.044+0000 7f18cc8c0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c80afdd0 con 0x7f18c8149dc0 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.044+0000 7f18cc8c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.044+0000 7f18cc8c0640 10 monclient(hunting): authenticate will time out at 1342.170297s 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.046+0000 7f18c5d9b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f18c8149dc0 0x7f18c814a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:59338/0 (socket says 172.21.15.155:59338) 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.046+0000 7f18c5d9b640 1 -- 172.21.15.155:0/750967918 learned_addr learned my addr 172.21.15.155:0/750967918 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.046+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855308350 0 0) 0x7f18c80afdd0 con 0x7f18c8149dc0 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.046+0000 7f18c559a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.046+0000 7f18c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.046+0000 7f18c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.046+0000 7f18c559a640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.046+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18b0003520 con 0x7f18c8149dc0 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441514619 0 0) 0x7f18b0003520 con 0x7f18c8149dc0 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 >> v1:172.21.15.183:6789/0 conn(0x7f18c80a9930 legacy=0x7f18c8143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 >> v1:172.21.15.155:6789/0 conn(0x7f18c814ff40 legacy=0x7f18c8152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 10 monclient: _finish_hunting 0 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 1 monclient: found mon.noname-b 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c814a950 con 0x7f18c8149dc0 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 10 monclient: _finish_auth 0 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 10 monclient: _check_auth_tickets 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18cc8c0640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18cc8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.047+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f18c0001a10 con 0x7f18c8149dc0 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 10 monclient: dump: 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:40.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18cc8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c5d9b640 1 -- 172.21.15.155:0/750967918 >> v1:172.21.15.155:6790/0 conn(0x7f18c8149dc0 legacy=0x7f18c814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c5d9b640 1 -- 172.21.15.155:0/750967918 >> v1:172.21.15.155:6790/0 conn(0x7f18c8149dc0 legacy=0x7f18c814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c5d9b640 1 --1- 172.21.15.155:0/750967918 >> v1:172.21.15.155:6790/0 conn(0x7f18c8149dc0 0x7f18c814a240 :-1 s=OPENED pgs=813387 cs=1 l=1).handle_message read tag failed 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c5d9b640 1 --1- 172.21.15.155:0/750967918 >> v1:172.21.15.155:6790/0 conn(0x7f18c8149dc0 0x7f18c814a240 :-1 s=OPENED pgs=813387 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 10 monclient: ms_handle_reset current mon v1:172.21.15.155:6790/0 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 >> v1:172.21.15.155:6790/0 conn(0x7f18c8149dc0 legacy=0x7f18c814a240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 10 monclient(hunting): picked mon.b con 0x7f18c80a9930 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 10 monclient(hunting): picked mon.a con 0x7f18c814ff40 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 10 monclient(hunting): picked mon.c con 0x7f18b000b090 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:40.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f18b0003520 con 0x7f18c814ff40 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f18c80aae60 con 0x7f18c80a9930 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f18c80b4310 con 0x7f18b000b090 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.048+0000 7f18c559a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.155:6790/0 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.051+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276436501 0 0) 0x7f18b0003520 con 0x7f18c814ff40 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.051+0000 7f18c559a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.051+0000 7f18c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.051+0000 7f18c559a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.051+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 --> v1:172.21.15.155:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f18b0010e40 con 0x7f18c814ff40 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.051+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558460980 0 0) 0x7f18c80b4310 con 0x7f18b000b090 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:40.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:42:40.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 --> v1:172.21.15.155:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f18b0003520 con 0x7f18b000b090 2024-03-29T11:42:40.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 224603233 0 0) 0x7f18b0010e40 con 0x7f18c814ff40 2024-03-29T11:42:40.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 >> v1:172.21.15.155:6790/0 conn(0x7f18b000b090 legacy=0x7f18b000d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 >> v1:172.21.15.183:6789/0 conn(0x7f18c80a9930 legacy=0x7f18b0003ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:40.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 10 monclient: _finish_hunting 0 2024-03-29T11:42:40.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 1 monclient: found mon.a 2024-03-29T11:42:40.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:40.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:42:40.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f18c80afdd0 con 0x7f18c814ff40 2024-03-29T11:42:40.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 10 monclient: _finish_auth 0 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 10 monclient: _check_auth_tickets 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f18c00020e0 con 0x7f18c814ff40 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 10 monclient: dump: 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:40.059 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18cc8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 1 -- 172.21.15.155:0/750967918 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18c0015e40 con 0x7f18c814ff40 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18cc8c0640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.052+0000 7f18cc8c0640 4 set_mon_vals no callback set 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.053+0000 7f18cc8c0640 10 monclient: shutdown 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.053+0000 7f18cc8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:40.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.053+0000 7f18cc8c0640 1 -- 172.21.15.155:0/750967918 >> v1:172.21.15.155:6789/0 conn(0x7f18c814ff40 legacy=0x7f18b0009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.053+0000 7f18cc8c0640 1 -- 172.21.15.155:0/750967918 shutdown_connections 2024-03-29T11:42:40.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.053+0000 7f18cc8c0640 1 -- 172.21.15.155:0/750967918 >> 172.21.15.155:0/750967918 conn(0x7f18c80b35e0 msgr2=0x7f18c80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:40.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.053+0000 7f18cc8c0640 1 -- 172.21.15.155:0/750967918 shutdown_connections 2024-03-29T11:42:40.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.053+0000 7f18cc8c0640 1 -- 172.21.15.155:0/750967918 wait complete. 2024-03-29T11:42:40.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.055+0000 7f18cc8c0640 10 monclient: build_initial_monmap 2024-03-29T11:42:40.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.055+0000 7f18cc8c0640 10 monclient: monmap: 2024-03-29T11:42:40.069 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:40.069 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:40.069 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:42:40.069 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.055+0000 7f18cc8c0640 1 librados: starting msgr at 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.055+0000 7f18cc8c0640 1 librados: starting objecter 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.055+0000 7f18cc8c0640 1 Processor -- start 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.055+0000 7f18cc8c0640 1 -- start start 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.055+0000 7f18cc8c0640 1 librados: setting wanted keys 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.055+0000 7f18cc8c0640 1 librados: calling monclient init 2024-03-29T11:42:40.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.055+0000 7f18cc8c0640 10 monclient: init 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.056+0000 7f18cc8c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.056+0000 7f18cc8c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.056+0000 7f18cc8c0640 10 monclient(hunting): picked mon.noname-c con 0x7f18b000b090 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.056+0000 7f18cc8c0640 10 monclient(hunting): picked mon.noname-b con 0x7f18c80a9930 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.056+0000 7f18cc8c0640 10 monclient(hunting): picked mon.noname-a con 0x7f18c814ff40 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.056+0000 7f18cc8c0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c814aae0 con 0x7f18c814ff40 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.056+0000 7f18cc8c0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c8144510 con 0x7f18c80a9930 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.056+0000 7f18cc8c0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c8146d60 con 0x7f18b000b090 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.056+0000 7f18cc8c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.056+0000 7f18cc8c0640 10 monclient(hunting): authenticate will time out at 1342.182744s 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.057+0000 7f18c659c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f18b000b090 0x7f18c81313e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33466/0 (socket says 172.21.15.155:33466) 2024-03-29T11:42:40.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.057+0000 7f18c659c640 1 -- 172.21.15.155:0/2294157099 learned_addr learned my addr 172.21.15.155:0/2294157099 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.058+0000 7f18b6ffd640 10 client.?.objecter ms_handle_connect 0x7f18b000b090 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.058+0000 7f18b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.058+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540014521 0 0) 0x7f18c8146d60 con 0x7f18b000b090 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.058+0000 7f18b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.058+0000 7f18b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.058+0000 7f18b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.058+0000 7f18b6ffd640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.058+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a8004eb0 con 0x7f18b000b090 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.061+0000 7f18b6ffd640 10 client.?.objecter ms_handle_connect 0x7f18c80a9930 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.061+0000 7f18b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.061+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 71243130 0 0) 0x7f18a8004eb0 con 0x7f18b000b090 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.061+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18c8146d60 con 0x7f18b000b090 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.062+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344724936 0 0) 0x7f18c8144510 con 0x7f18c80a9930 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.062+0000 7f18b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.062+0000 7f18b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.062+0000 7f18b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.062+0000 7f18b6ffd640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.062+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a8005590 con 0x7f18c80a9930 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.062+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f18bc001970 con 0x7f18b000b090 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.062+0000 7f18b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.062+0000 7f18b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.062+0000 7f18b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:40.073 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.062+0000 7f18b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 10 client.?.objecter ms_handle_connect 0x7f18c814ff40 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2103185603 0 0) 0x7f18a8005590 con 0x7f18c80a9930 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18c8144510 con 0x7f18c80a9930 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f18c0015c00 con 0x7f18c80a9930 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:40.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1561014816 0 0) 0x7f18c8146d60 con 0x7f18b000b090 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 >> v1:172.21.15.183:6789/0 conn(0x7f18c80a9930 legacy=0x7f18c812fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 >> v1:172.21.15.155:6789/0 conn(0x7f18c814ff40 legacy=0x7f18c8130420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 1 monclient: found mon.c 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c8147460 con 0x7f18b000b090 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18cc8c0640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18cc8c0640 10 monclient: _renew_subs 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18cc8c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.063+0000 7f18cc8c0640 1 -- 172.21.15.155:0/2294157099 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c8147710 con 0x7f18b000b090 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.064+0000 7f18cc8c0640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.064+0000 7f18cc8c0640 10 monclient: _renew_subs 2024-03-29T11:42:40.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.064+0000 7f18cc8c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.064+0000 7f18cc8c0640 1 -- 172.21.15.155:0/2294157099 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c81de4f0 con 0x7f18b000b090 2024-03-29T11:42:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.064+0000 7f18cc8c0640 1 librados: init done 2024-03-29T11:42:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.065+0000 7f18cc8c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.065+0000 7f18cc8c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.065+0000 7f18cc8c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.066+0000 7f18cc8c0640 1 -- 172.21.15.155:0/2294157099 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c8131fe0 con 0x7f18b000b090 2024-03-29T11:42:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.066+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18bc003610 con 0x7f18b000b090 2024-03-29T11:42:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.066+0000 7f18b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.066+0000 7f18c4d99640 4 set_mon_vals no callback set 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.067+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f18bc001970 con 0x7f18b000b090 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.067+0000 7f18b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.067+0000 7f18b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.067+0000 7f18b6ffd640 10 monclient: dump: 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:40.079 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:40.080 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:40.080 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:40.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.068+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f18bc0038c0 con 0x7f18b000b090 2024-03-29T11:42:40.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.069+0000 7f18c5d9b640 10 monclient: get_auth_request con 0x7f18a806e320 auth_method 0 2024-03-29T11:42:40.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.069+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f18bc054550 con 0x7f18b000b090 2024-03-29T11:42:40.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.069+0000 7f18b6ffd640 10 client.4562.objecter ms_dispatch 0x7f18c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:40.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.069+0000 7f18b6ffd640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:40.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.069+0000 7f18b6ffd640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:40.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.069+0000 7f18b6ffd640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T11:42:40.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.071+0000 7f18b6ffd640 10 client.4562.objecter ms_handle_connect 0x7f18a806e320 2024-03-29T11:42:40.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.080+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18bc050810 con 0x7f18b000b090 2024-03-29T11:42:40.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.080+0000 7f18b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:40.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.080+0000 7f18b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:40.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.399+0000 7f18cc8c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:40.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.399+0000 7f18cc8c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:40.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.399+0000 7f18cc8c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:40.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.399+0000 7f18cc8c0640 1 -- 172.21.15.155:0/2294157099 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18c8075f10 con 0x7f18b000b090 2024-03-29T11:42:40.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.401+0000 7f18b6ffd640 1 -- 172.21.15.155:0/2294157099 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f18bc050810 con 0x7f18b000b090 2024-03-29T11:42:40.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.401+0000 7f18b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:40.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.401+0000 7f18b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:40.404 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:42:40.404 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:40.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.405+0000 7f18cc8c0640 10 librados: watch_flush enter 2024-03-29T11:42:40.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.405+0000 7f18cc8c0640 10 librados: watch_flush exit 2024-03-29T11:42:40.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.405+0000 7f18cc8c0640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T11:42:40.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.405+0000 7f18cc8c0640 10 client.4562.objecter successfully canceled tick 2024-03-29T11:42:40.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.407+0000 7f18cc8c0640 1 -- 172.21.15.155:0/2294157099 >> v1:172.21.15.155:6816/1760814708 conn(0x7f18a806e320 legacy=0x7f18a80707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.407+0000 7f18cc8c0640 10 monclient: shutdown 2024-03-29T11:42:40.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.407+0000 7f18cc8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:40.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.407+0000 7f18cc8c0640 1 -- 172.21.15.155:0/2294157099 >> v1:172.21.15.155:6790/0 conn(0x7f18b000b090 legacy=0x7f18c81313e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.407+0000 7f18cc8c0640 1 -- 172.21.15.155:0/2294157099 shutdown_connections 2024-03-29T11:42:40.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.407+0000 7f18cc8c0640 1 -- 172.21.15.155:0/2294157099 >> 172.21.15.155:0/2294157099 conn(0x7f18c80b35e0 msgr2=0x7f18c813d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:40.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.409+0000 7f18cc8c0640 1 -- 172.21.15.155:0/2294157099 shutdown_connections 2024-03-29T11:42:40.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.409+0000 7f18cc8c0640 1 -- 172.21.15.155:0/2294157099 wait complete. 2024-03-29T11:42:40.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:40.409+0000 7f18cc8c0640 1 librados: shutdown 2024-03-29T11:42:40.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:47.236 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:47.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.375+0000 7f3c70c5f640 10 monclient: get_monmap_and_config 2024-03-29T11:42:47.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.375+0000 7f3c70c5f640 10 monclient: build_initial_monmap 2024-03-29T11:42:47.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.375+0000 7f3c70c5f640 10 monclient: monmap: 2024-03-29T11:42:47.379 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:47.379 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:47.379 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:42:47.376667+0000 2024-03-29T11:42:47.379 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:42:47.376667+0000 2024-03-29T11:42:47.379 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:47.379 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:47.379 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:47.379 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:42:47.380 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:42:47.380 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:47.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.377+0000 7f3c70c5f640 1 Processor -- start 2024-03-29T11:42:47.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.377+0000 7f3c70c5f640 1 -- start start 2024-03-29T11:42:47.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.377+0000 7f3c70c5f640 10 monclient: init 2024-03-29T11:42:47.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.378+0000 7f3c70c5f640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:47.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.378+0000 7f3c70c5f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:47.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.378+0000 7f3c70c5f640 10 monclient(hunting): picked mon.noname-a con 0x7f3c64006e30 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:47.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.378+0000 7f3c70c5f640 10 monclient(hunting): picked mon.noname-b con 0x7f3c64035e00 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:47.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.378+0000 7f3c70c5f640 10 monclient(hunting): picked mon.noname-c con 0x7f3c640c9670 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:47.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.378+0000 7f3c70c5f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64035970 con 0x7f3c64006e30 2024-03-29T11:42:47.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.378+0000 7f3c70c5f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64036280 con 0x7f3c640c9670 2024-03-29T11:42:47.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.378+0000 7f3c70c5f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64035640 con 0x7f3c64035e00 2024-03-29T11:42:47.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.378+0000 7f3c70c5f640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:47.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.378+0000 7f3c70c5f640 10 monclient(hunting): authenticate will time out at 1349.505590s 2024-03-29T11:42:47.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.379+0000 7f3c6affd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f3c64035e00 0x7f3c640c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36562/0 (socket says 172.21.15.155:36562) 2024-03-29T11:42:47.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.379+0000 7f3c6affd640 1 -- 172.21.15.155:0/4047856728 learned_addr learned my addr 172.21.15.155:0/4047856728 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:47.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.380+0000 7f3c6a7fc640 1 -- 172.21.15.155:0/4047856728 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622531863 0 0) 0x7f3c64035640 con 0x7f3c64035e00 2024-03-29T11:42:47.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.380+0000 7f3c6a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:47.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.380+0000 7f3c6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:47.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.380+0000 7f3c6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:47.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.380+0000 7f3c6a7fc640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:42:47.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.380+0000 7f3c6a7fc640 1 -- 172.21.15.155:0/4047856728 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5c003520 con 0x7f3c64035e00 2024-03-29T11:42:47.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.381+0000 7f3c6a7fc640 1 -- 172.21.15.155:0/4047856728 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2393285304 0 0) 0x7f3c5c003520 con 0x7f3c64035e00 2024-03-29T11:42:47.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.381+0000 7f3c6a7fc640 1 -- 172.21.15.155:0/4047856728 >> v1:172.21.15.183:6789/0 conn(0x7f3c640c9670 legacy=0x7f3c640d7560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:47.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.382+0000 7f3c6a7fc640 1 -- 172.21.15.155:0/4047856728 >> v1:172.21.15.155:6789/0 conn(0x7f3c64006e30 legacy=0x7f3c640cbcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:47.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:47.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:42:47.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:47.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:42:47.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 1 -- 172.21.15.155:0/4047856728 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c64034e20 con 0x7f3c64035e00 2024-03-29T11:42:47.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:47.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:47.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:47.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 1 -- 172.21.15.155:0/4047856728 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3c60001690 con 0x7f3c64035e00 2024-03-29T11:42:47.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:47.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:47.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c6a7fc640 10 monclient: dump: 2024-03-29T11:42:47.389 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:47.389 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:47.389 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:47.389 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:47.389 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:47.389 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:47.390 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:47.390 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:47.390 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:47.390 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:47.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c70c5f640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:42:47.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.384+0000 7f3c70c5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:47.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.385+0000 7f3c6a7fc640 1 -- 172.21.15.155:0/4047856728 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c600018f0 con 0x7f3c64035e00 2024-03-29T11:42:47.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.385+0000 7f3c6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:47.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.385+0000 7f3c6a7fc640 1 -- 172.21.15.155:0/4047856728 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3c60001d40 con 0x7f3c64035e00 2024-03-29T11:42:47.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.385+0000 7f3c70c5f640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:47.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.385+0000 7f3c70c5f640 4 set_mon_vals no callback set 2024-03-29T11:42:47.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.386+0000 7f3c6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:47.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.386+0000 7f3c6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:47.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.386+0000 7f3c6a7fc640 10 monclient: dump: 2024-03-29T11:42:47.393 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:47.393 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:47.393 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:47.393 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:47.393 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:47.393 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:47.393 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:47.393 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:47.394 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:47.394 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:47.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.386+0000 7f3c70c5f640 10 monclient: shutdown 2024-03-29T11:42:47.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.386+0000 7f3c70c5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:47.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.386+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4047856728 >> v1:172.21.15.155:6790/0 conn(0x7f3c64035e00 legacy=0x7f3c640c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:47.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.387+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4047856728 shutdown_connections 2024-03-29T11:42:47.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.387+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4047856728 >> 172.21.15.155:0/4047856728 conn(0x7f3c64005c00 msgr2=0x7f3c64006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:47.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.387+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4047856728 shutdown_connections 2024-03-29T11:42:47.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.387+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4047856728 wait complete. 2024-03-29T11:42:47.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.388+0000 7f3c70c5f640 10 monclient: build_initial_monmap 2024-03-29T11:42:47.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.388+0000 7f3c70c5f640 10 monclient: monmap: 2024-03-29T11:42:47.395 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:47.395 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:47.395 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.389+0000 7f3c70c5f640 1 librados: starting msgr at 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.389+0000 7f3c70c5f640 1 librados: starting objecter 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.389+0000 7f3c70c5f640 1 Processor -- start 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.389+0000 7f3c70c5f640 1 -- start start 2024-03-29T11:42:47.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.389+0000 7f3c70c5f640 1 librados: setting wanted keys 2024-03-29T11:42:47.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.389+0000 7f3c70c5f640 1 librados: calling monclient init 2024-03-29T11:42:47.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.389+0000 7f3c70c5f640 10 monclient: init 2024-03-29T11:42:47.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.390+0000 7f3c70c5f640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:47.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.390+0000 7f3c70c5f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:47.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.390+0000 7f3c70c5f640 10 monclient(hunting): picked mon.noname-c con 0x7f3c64006e30 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:47.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.390+0000 7f3c70c5f640 10 monclient(hunting): picked mon.noname-b con 0x7f3c64035e00 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:47.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.390+0000 7f3c70c5f640 10 monclient(hunting): picked mon.noname-a con 0x7f3c640c9670 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:47.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.390+0000 7f3c70c5f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640cd860 con 0x7f3c640c9670 2024-03-29T11:42:47.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.390+0000 7f3c70c5f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640cdb40 con 0x7f3c64035e00 2024-03-29T11:42:47.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.390+0000 7f3c70c5f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640c5110 con 0x7f3c64006e30 2024-03-29T11:42:47.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.390+0000 7f3c70c5f640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:47.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.390+0000 7f3c70c5f640 10 monclient(hunting): authenticate will time out at 1349.517470s 2024-03-29T11:42:47.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.391+0000 7f3c6b7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f3c64006e30 0x7f3c640c4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:53990/0 (socket says 172.21.15.155:53990) 2024-03-29T11:42:47.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.391+0000 7f3c6b7fe640 1 -- 172.21.15.155:0/4065094227 learned_addr learned my addr 172.21.15.155:0/4065094227 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:47.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.392+0000 7f3c57fff640 10 client.?.objecter ms_handle_connect 0x7f3c64006e30 2024-03-29T11:42:47.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.392+0000 7f3c57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:47.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.392+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514915641 0 0) 0x7f3c640c5110 con 0x7f3c64006e30 2024-03-29T11:42:47.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.392+0000 7f3c57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:47.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.392+0000 7f3c57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:47.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.392+0000 7f3c57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:47.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.392+0000 7f3c57fff640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:42:47.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.392+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c50004eb0 con 0x7f3c64006e30 2024-03-29T11:42:47.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.393+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 66210803 0 0) 0x7f3c50004eb0 con 0x7f3c64006e30 2024-03-29T11:42:47.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.393+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c640c5110 con 0x7f3c64006e30 2024-03-29T11:42:47.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.393+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3c6c04fdd0 con 0x7f3c64006e30 2024-03-29T11:42:47.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.393+0000 7f3c57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.393+0000 7f3c57fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.393+0000 7f3c57fff640 10 monclient(hunting): dump: 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:47.405 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.393+0000 7f3c57fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.393+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 898392534 0 0) 0x7f3c640c5110 con 0x7f3c64006e30 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.393+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 >> v1:172.21.15.183:6789/0 conn(0x7f3c64035e00 legacy=0x7f3c640b9ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.394+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 >> v1:172.21.15.155:6789/0 conn(0x7f3c640c9670 legacy=0x7f3c6402d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c57fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c57fff640 1 monclient: found mon.c 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c57fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c640ba1c0 con 0x7f3c64006e30 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c57fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c57fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:47.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c70c5f640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c70c5f640 10 monclient: _renew_subs 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c70c5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4065094227 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c640ba410 con 0x7f3c64006e30 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c70c5f640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c70c5f640 10 monclient: _renew_subs 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c70c5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4065094227 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c64033370 con 0x7f3c64006e30 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.395+0000 7f3c70c5f640 1 librados: init done 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c57fff640 10 client.4568.objecter ms_handle_connect 0x7f3c640c9670 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c57fff640 10 client.4568.objecter resend_mon_ops 2024-03-29T11:42:47.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c6c04f400 con 0x7f3c64006e30 2024-03-29T11:42:47.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:47.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3c6c075f10 con 0x7f3c64006e30 2024-03-29T11:42:47.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:47.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:47.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c57fff640 10 monclient: dump: 2024-03-29T11:42:47.408 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c70c5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c70c5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c70c5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:47.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.396+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4065094227 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c64007b90 con 0x7f3c64006e30 2024-03-29T11:42:47.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.398+0000 7f3c697fa640 4 set_mon_vals no callback set 2024-03-29T11:42:47.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.398+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f3c6c067bf0 con 0x7f3c64006e30 2024-03-29T11:42:47.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.399+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f3c6c0ff5d0 con 0x7f3c64006e30 2024-03-29T11:42:47.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.399+0000 7f3c57fff640 10 client.4568.objecter ms_dispatch 0x7f3c6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:47.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.399+0000 7f3c6affd640 10 monclient: get_auth_request con 0x7f3c5006dad0 auth_method 0 2024-03-29T11:42:47.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.399+0000 7f3c57fff640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:47.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.399+0000 7f3c57fff640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:47.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.399+0000 7f3c57fff640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T11:42:47.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.400+0000 7f3c57fff640 10 client.4568.objecter ms_handle_connect 0x7f3c5006dad0 2024-03-29T11:42:47.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.408+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c6c0caeb0 con 0x7f3c64006e30 2024-03-29T11:42:47.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.408+0000 7f3c57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:47.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.408+0000 7f3c57fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:47.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.736+0000 7f3c70c5f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:47.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.736+0000 7f3c70c5f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:47.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.736+0000 7f3c70c5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:42:47.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.736+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4065094227 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c640c52a0 con 0x7f3c64006e30 2024-03-29T11:42:47.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.738+0000 7f3c57fff640 1 -- 172.21.15.155:0/4065094227 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f3c6c0ccde0 con 0x7f3c64006e30 2024-03-29T11:42:47.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.738+0000 7f3c57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:47.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.738+0000 7f3c57fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:47.742 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:42:47.743 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.743+0000 7f3c70c5f640 10 librados: watch_flush enter 2024-03-29T11:42:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.743+0000 7f3c70c5f640 10 librados: watch_flush exit 2024-03-29T11:42:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.743+0000 7f3c70c5f640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T11:42:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.743+0000 7f3c70c5f640 10 client.4568.objecter successfully canceled tick 2024-03-29T11:42:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.744+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4065094227 >> v1:172.21.15.155:6816/1760814708 conn(0x7f3c5006dad0 legacy=0x7f3c5006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.744+0000 7f3c70c5f640 10 monclient: shutdown 2024-03-29T11:42:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.744+0000 7f3c70c5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.744+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4065094227 >> v1:172.21.15.155:6790/0 conn(0x7f3c64006e30 legacy=0x7f3c640c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:47.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.745+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4065094227 shutdown_connections 2024-03-29T11:42:47.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.745+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4065094227 >> 172.21.15.155:0/4065094227 conn(0x7f3c64005c00 msgr2=0x7f3c64039350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:47.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.745+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4065094227 shutdown_connections 2024-03-29T11:42:47.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.745+0000 7f3c70c5f640 1 -- 172.21.15.155:0/4065094227 wait complete. 2024-03-29T11:42:47.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:47.745+0000 7f3c70c5f640 1 librados: shutdown 2024-03-29T11:42:47.768 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:54.574 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:54.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.728+0000 7f4f31396640 10 monclient: get_monmap_and_config 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.728+0000 7f4f31396640 10 monclient: build_initial_monmap 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.728+0000 7f4f31396640 10 monclient: monmap: 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:42:54.729703+0000 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:42:54.729703+0000 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:42:54.733 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:54.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.731+0000 7f4f31396640 1 Processor -- start 2024-03-29T11:42:54.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.731+0000 7f4f31396640 1 -- start start 2024-03-29T11:42:54.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.731+0000 7f4f31396640 10 monclient: init 2024-03-29T11:42:54.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.732+0000 7f4f31396640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:54.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.732+0000 7f4f31396640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:54.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.732+0000 7f4f31396640 10 monclient(hunting): picked mon.noname-a con 0x7f4f2c0b4100 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:54.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.732+0000 7f4f31396640 10 monclient(hunting): picked mon.noname-c con 0x7f4f2c13ee90 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:54.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.732+0000 7f4f31396640 10 monclient(hunting): picked mon.noname-b con 0x7f4f2c0af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:54.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.732+0000 7f4f31396640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c0afd10 con 0x7f4f2c0b4100 2024-03-29T11:42:54.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.732+0000 7f4f31396640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c0af6f0 con 0x7f4f2c13ee90 2024-03-29T11:42:54.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.732+0000 7f4f31396640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c0aaf60 con 0x7f4f2c0af900 2024-03-29T11:42:54.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.732+0000 7f4f31396640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:54.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.732+0000 7f4f31396640 10 monclient(hunting): authenticate will time out at 1356.859761s 2024-03-29T11:42:54.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.735+0000 7f4f2affd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f4f2c0b4100 0x7f4f2c0a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:41357/0 (socket says 172.21.15.155:41357) 2024-03-29T11:42:54.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.735+0000 7f4f2affd640 1 -- 172.21.15.155:0/2223974987 learned_addr learned my addr 172.21.15.155:0/2223974987 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:54.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.740+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489743903 0 0) 0x7f4f2c0afd10 con 0x7f4f2c0b4100 2024-03-29T11:42:54.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.740+0000 7f4f29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:54.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.740+0000 7f4f29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:54.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.740+0000 7f4f29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.740+0000 7f4f29ffb640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.740+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f14003520 con 0x7f4f2c0b4100 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303841292 0 0) 0x7f4f2c0aaf60 con 0x7f4f2c0af900 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f2c0afd10 con 0x7f4f2c0af900 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917596985 0 0) 0x7f4f2c0af6f0 con 0x7f4f2c13ee90 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:42:54.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.741+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f2c0aaf60 con 0x7f4f2c13ee90 2024-03-29T11:42:54.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1914931100 0 0) 0x7f4f14003520 con 0x7f4f2c0b4100 2024-03-29T11:42:54.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 >> v1:172.21.15.155:6790/0 conn(0x7f4f2c0af900 legacy=0x7f4f2c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:54.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 >> v1:172.21.15.183:6789/0 conn(0x7f4f2c13ee90 legacy=0x7f4f2c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:54.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:54.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 1 monclient: found mon.noname-a 2024-03-29T11:42:54.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:54.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:42:54.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c13e970 con 0x7f4f2c0b4100 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4f20001690 con 0x7f4f2c0b4100 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f29ffb640 10 monclient: dump: 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:54.748 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:54.749 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:54.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f31396640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:42:54.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.743+0000 7f4f31396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:54.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.744+0000 7f4f29ffb640 1 -- 172.21.15.155:0/2223974987 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f200017f0 con 0x7f4f2c0b4100 2024-03-29T11:42:54.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.744+0000 7f4f29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:54.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.744+0000 7f4f31396640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:54.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.744+0000 7f4f31396640 4 set_mon_vals no callback set 2024-03-29T11:42:54.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.745+0000 7f4f31396640 10 monclient: shutdown 2024-03-29T11:42:54.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.745+0000 7f4f31396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:54.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.745+0000 7f4f31396640 1 -- 172.21.15.155:0/2223974987 >> v1:172.21.15.155:6789/0 conn(0x7f4f2c0b4100 legacy=0x7f4f2c0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:54.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.745+0000 7f4f31396640 1 -- 172.21.15.155:0/2223974987 shutdown_connections 2024-03-29T11:42:54.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.745+0000 7f4f31396640 1 -- 172.21.15.155:0/2223974987 >> 172.21.15.155:0/2223974987 conn(0x7f4f2c0b35e0 msgr2=0x7f4f2c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:54.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.752+0000 7f4f31396640 1 -- 172.21.15.155:0/2223974987 shutdown_connections 2024-03-29T11:42:54.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.752+0000 7f4f31396640 1 -- 172.21.15.155:0/2223974987 wait complete. 2024-03-29T11:42:54.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.753+0000 7f4f31396640 10 monclient: build_initial_monmap 2024-03-29T11:42:54.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.753+0000 7f4f31396640 10 monclient: monmap: 2024-03-29T11:42:54.760 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:42:54.760 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:54.761 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:42:54.761 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:42:54.761 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:54.761 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:54.761 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:42:54.761 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:42:54.761 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:42:54.761 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:54.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.753+0000 7f4f31396640 1 librados: starting msgr at 2024-03-29T11:42:54.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.753+0000 7f4f31396640 1 librados: starting objecter 2024-03-29T11:42:54.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.753+0000 7f4f31396640 1 Processor -- start 2024-03-29T11:42:54.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 1 -- start start 2024-03-29T11:42:54.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 1 librados: setting wanted keys 2024-03-29T11:42:54.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 1 librados: calling monclient init 2024-03-29T11:42:54.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 10 monclient: init 2024-03-29T11:42:54.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:54.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:54.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 10 monclient(hunting): picked mon.noname-c con 0x7f4f2c0af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:42:54.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 10 monclient(hunting): picked mon.noname-a con 0x7f4f2c0b4100 addr v1:172.21.15.155:6789/0 2024-03-29T11:42:54.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 10 monclient(hunting): picked mon.noname-b con 0x7f4f2c13ee90 addr v1:172.21.15.183:6789/0 2024-03-29T11:42:54.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c141640 con 0x7f4f2c0b4100 2024-03-29T11:42:54.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c141900 con 0x7f4f2c13ee90 2024-03-29T11:42:54.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c141be0 con 0x7f4f2c0af900 2024-03-29T11:42:54.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:54.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.754+0000 7f4f31396640 10 monclient(hunting): authenticate will time out at 1356.881762s 2024-03-29T11:42:54.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.762+0000 7f4f2a7fc640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f4f2c0b4100 0x7f4f2c1e3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:42487/0 (socket says 172.21.15.155:42487) 2024-03-29T11:42:54.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.762+0000 7f4f2a7fc640 1 -- 172.21.15.155:0/106894460 learned_addr learned my addr 172.21.15.155:0/106894460 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:42:54.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 client.?.objecter ms_handle_connect 0x7f4f2c0b4100 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 client.?.objecter ms_handle_connect 0x7f4f2c13ee90 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061119301 0 0) 0x7f4f2c141640 con 0x7f4f2c0b4100 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f18004eb0 con 0x7f4f2c0b4100 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891293230 0 0) 0x7f4f2c141900 con 0x7f4f2c13ee90 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:54.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:42:54.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f2c141640 con 0x7f4f2c13ee90 2024-03-29T11:42:54.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.764+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2485602939 0 0) 0x7f4f18004eb0 con 0x7f4f2c0b4100 2024-03-29T11:42:54.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f2c141900 con 0x7f4f2c0b4100 2024-03-29T11:42:54.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 10 client.?.objecter ms_handle_connect 0x7f4f2c0af900 2024-03-29T11:42:54.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4f1c001970 con 0x7f4f2c0b4100 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 10 monclient(hunting): dump: 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:54.772 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 682711783 0 0) 0x7f4f2c141640 con 0x7f4f2c13ee90 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f18004eb0 con 0x7f4f2c13ee90 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4f24001c30 con 0x7f4f2c13ee90 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 10 monclient(hunting): dump: 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:54.773 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.765+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1060906309 0 0) 0x7f4f18004eb0 con 0x7f4f2c13ee90 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 >> v1:172.21.15.155:6790/0 conn(0x7f4f2c0af900 legacy=0x7f4f2c1e4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 >> v1:172.21.15.155:6789/0 conn(0x7f4f2c0b4100 legacy=0x7f4f2c1e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 1 monclient: found mon.b 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c142350 con 0x7f4f2c13ee90 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:54.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f31396640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f31396640 10 monclient: _renew_subs 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f31396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f31396640 1 -- 172.21.15.155:0/106894460 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c142000 con 0x7f4f2c13ee90 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f31396640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f31396640 10 monclient: _renew_subs 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f31396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f31396640 1 -- 172.21.15.155:0/106894460 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c1401e0 con 0x7f4f2c13ee90 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f31396640 1 librados: init done 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f240020c0 con 0x7f4f2c13ee90 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f4f24003b30 con 0x7f4f2c13ee90 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.766+0000 7f4f137fe640 10 monclient: dump: 2024-03-29T11:42:54.775 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:42:54.776 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:42:54.776 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:54.776 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:42:54.776 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:54.776 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:42:54.776 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:42:54.776 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:42:54.776 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:42:54.776 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:42:54.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.769+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f4f24019b00 con 0x7f4f2c13ee90 2024-03-29T11:42:54.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.769+0000 7f4f297fa640 4 set_mon_vals no callback set 2024-03-29T11:42:54.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.769+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f4f24083a70 con 0x7f4f2c13ee90 2024-03-29T11:42:54.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.769+0000 7f4f137fe640 10 client.4477.objecter ms_dispatch 0x7f4f2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:54.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.769+0000 7f4f137fe640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:54.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.769+0000 7f4f137fe640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:54.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.769+0000 7f4f137fe640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T11:42:54.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.770+0000 7f4f2affd640 10 monclient: get_auth_request con 0x7f4f1806dfe0 auth_method 0 2024-03-29T11:42:54.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.770+0000 7f4f31396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:54.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.774+0000 7f4f31396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:54.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.774+0000 7f4f31396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:54.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.774+0000 7f4f31396640 1 -- 172.21.15.155:0/106894460 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0c005080 con 0x7f4f2c13ee90 2024-03-29T11:42:54.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.775+0000 7f4f137fe640 10 client.4477.objecter ms_handle_connect 0x7f4f1806dfe0 2024-03-29T11:42:54.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.781+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f2404f180 con 0x7f4f2c13ee90 2024-03-29T11:42:54.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.781+0000 7f4f137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:54.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:54.781+0000 7f4f137fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:55.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.121+0000 7f4f31396640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:55.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.121+0000 7f4f31396640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:55.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.121+0000 7f4f31396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:42:55.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.121+0000 7f4f31396640 1 -- 172.21.15.155:0/106894460 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f0c005330 con 0x7f4f2c13ee90 2024-03-29T11:42:55.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.123+0000 7f4f137fe640 1 -- 172.21.15.155:0/106894460 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f4f24051280 con 0x7f4f2c13ee90 2024-03-29T11:42:55.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.123+0000 7f4f137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:55.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.123+0000 7f4f137fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:55.128 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:42:55.128 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:55.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.127+0000 7f4f31396640 10 librados: watch_flush enter 2024-03-29T11:42:55.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.127+0000 7f4f31396640 10 librados: watch_flush exit 2024-03-29T11:42:55.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.127+0000 7f4f31396640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T11:42:55.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.127+0000 7f4f31396640 10 client.4477.objecter successfully canceled tick 2024-03-29T11:42:55.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.127+0000 7f4f31396640 1 -- 172.21.15.155:0/106894460 >> v1:172.21.15.155:6816/1760814708 conn(0x7f4f1806dfe0 legacy=0x7f4f180704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:55.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.127+0000 7f4f31396640 10 monclient: shutdown 2024-03-29T11:42:55.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.128+0000 7f4f31396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:55.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.128+0000 7f4f31396640 1 -- 172.21.15.155:0/106894460 >> v1:172.21.15.183:6789/0 conn(0x7f4f2c13ee90 legacy=0x7f4f2c1e3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:55.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.128+0000 7f4f31396640 1 -- 172.21.15.155:0/106894460 shutdown_connections 2024-03-29T11:42:55.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.128+0000 7f4f31396640 1 -- 172.21.15.155:0/106894460 >> 172.21.15.155:0/106894460 conn(0x7f4f2c0b35e0 msgr2=0x7f4f2c0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:55.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.129+0000 7f4f31396640 1 -- 172.21.15.155:0/106894460 shutdown_connections 2024-03-29T11:42:55.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.129+0000 7f4f31396640 1 -- 172.21.15.155:0/106894460 wait complete. 2024-03-29T11:42:55.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:42:55.130+0000 7f4f31396640 1 librados: shutdown 2024-03-29T11:42:55.152 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:01.959 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:02.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.114+0000 7f7fa17a5640 10 monclient: get_monmap_and_config 2024-03-29T11:43:02.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.114+0000 7f7fa17a5640 10 monclient: build_initial_monmap 2024-03-29T11:43:02.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.114+0000 7f7fa17a5640 10 monclient: monmap: 2024-03-29T11:43:02.119 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:02.119 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:02.119 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:43:02.116191+0000 2024-03-29T11:43:02.119 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:43:02.116191+0000 2024-03-29T11:43:02.119 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:02.120 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:02.120 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:02.120 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:43:02.120 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:43:02.120 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:02.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.118+0000 7f7fa17a5640 1 Processor -- start 2024-03-29T11:43:02.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.122+0000 7f7fa17a5640 1 -- start start 2024-03-29T11:43:02.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.122+0000 7f7fa17a5640 10 monclient: init 2024-03-29T11:43:02.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.123+0000 7f7fa17a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:02.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.123+0000 7f7fa17a5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:02.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.123+0000 7f7fa17a5640 10 monclient(hunting): picked mon.noname-b con 0x7f7f9c0a9ac0 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:02.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.123+0000 7f7fa17a5640 10 monclient(hunting): picked mon.noname-c con 0x7f7f9c13ee90 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:02.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.123+0000 7f7fa17a5640 10 monclient(hunting): picked mon.noname-a con 0x7f7f9c0af900 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:02.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.123+0000 7f7fa17a5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9c0afd10 con 0x7f7f9c0af900 2024-03-29T11:43:02.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.123+0000 7f7fa17a5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9c0af6f0 con 0x7f7f9c13ee90 2024-03-29T11:43:02.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.123+0000 7f7fa17a5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9c0aac80 con 0x7f7f9c0a9ac0 2024-03-29T11:43:02.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.123+0000 7f7fa17a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:02.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.123+0000 7f7fa17a5640 10 monclient(hunting): authenticate will time out at 1364.250437s 2024-03-29T11:43:02.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.126+0000 7f7f9b7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7f9c0af900 0x7f7f9c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:53434/0 (socket says 172.21.15.155:53434) 2024-03-29T11:43:02.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.126+0000 7f7f9b7fe640 1 -- 172.21.15.155:0/2815341924 learned_addr learned my addr 172.21.15.155:0/2815341924 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:02.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.129+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581166987 0 0) 0x7f7f9c0af6f0 con 0x7f7f9c13ee90 2024-03-29T11:43:02.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.129+0000 7f7f99ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:02.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.129+0000 7f7f99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:02.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.129+0000 7f7f99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:02.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.129+0000 7f7f99ffb640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:43:02.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.129+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f84003520 con 0x7f7f9c13ee90 2024-03-29T11:43:02.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011520991 0 0) 0x7f7f9c0afd10 con 0x7f7f9c0af900 2024-03-29T11:43:02.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:02.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:02.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:02.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:43:02.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f9c0af6f0 con 0x7f7f9c0af900 2024-03-29T11:43:02.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677075942 0 0) 0x7f7f9c0aac80 con 0x7f7f9c0a9ac0 2024-03-29T11:43:02.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:02.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:02.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:02.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:43:02.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f9c0afd10 con 0x7f7f9c0a9ac0 2024-03-29T11:43:02.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1385167318 0 0) 0x7f7f84003520 con 0x7f7f9c13ee90 2024-03-29T11:43:02.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 >> v1:172.21.15.155:6790/0 conn(0x7f7f9c0a9ac0 legacy=0x7f7f9c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 >> v1:172.21.15.155:6789/0 conn(0x7f7f9c0af900 legacy=0x7f7f9c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:43:02.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 1 monclient: found mon.noname-c 2024-03-29T11:43:02.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:02.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:43:02.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c0aaf60 con 0x7f7f9c13ee90 2024-03-29T11:43:02.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient: _finish_auth 0 2024-03-29T11:43:02.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:43:02.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:02.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7fa17a5640 5 monclient: authenticate success, global_id 4480 2024-03-29T11:43:02.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7fa17a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:02.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7f940018c0 con 0x7f7f9c13ee90 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7f99ffb640 10 monclient: dump: 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:02.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.130+0000 7f7fa17a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.131+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f94001a20 con 0x7f7f9c13ee90 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.131+0000 7f7f99ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.131+0000 7f7fa17a5640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.131+0000 7f7fa17a5640 4 set_mon_vals no callback set 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.132+0000 7f7f99ffb640 1 -- 172.21.15.155:0/2815341924 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7f94001e30 con 0x7f7f9c13ee90 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.132+0000 7f7f99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.132+0000 7f7f99ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.132+0000 7f7f99ffb640 10 monclient: dump: 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:02.139 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.132+0000 7f7fa17a5640 10 monclient: shutdown 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.132+0000 7f7fa17a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.132+0000 7f7fa17a5640 1 -- 172.21.15.155:0/2815341924 >> v1:172.21.15.183:6789/0 conn(0x7f7f9c13ee90 legacy=0x7f7f9c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.132+0000 7f7fa17a5640 1 -- 172.21.15.155:0/2815341924 shutdown_connections 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.132+0000 7f7fa17a5640 1 -- 172.21.15.155:0/2815341924 >> 172.21.15.155:0/2815341924 conn(0x7f7f9c0b35e0 msgr2=0x7f7f9c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.133+0000 7f7fa17a5640 1 -- 172.21.15.155:0/2815341924 shutdown_connections 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.133+0000 7f7fa17a5640 1 -- 172.21.15.155:0/2815341924 wait complete. 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.134+0000 7f7fa17a5640 10 monclient: build_initial_monmap 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.134+0000 7f7fa17a5640 10 monclient: monmap: 2024-03-29T11:43:02.140 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:02.141 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:02.141 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:43:02.141 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:43:02.141 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:02.141 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:02.141 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:02.141 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:43:02.141 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:43:02.141 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:02.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.135+0000 7f7fa17a5640 1 librados: starting msgr at 2024-03-29T11:43:02.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.135+0000 7f7fa17a5640 1 librados: starting objecter 2024-03-29T11:43:02.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.135+0000 7f7fa17a5640 1 Processor -- start 2024-03-29T11:43:02.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.135+0000 7f7fa17a5640 1 -- start start 2024-03-29T11:43:02.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.135+0000 7f7fa17a5640 1 librados: setting wanted keys 2024-03-29T11:43:02.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.135+0000 7f7fa17a5640 1 librados: calling monclient init 2024-03-29T11:43:02.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.135+0000 7f7fa17a5640 10 monclient: init 2024-03-29T11:43:02.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.136+0000 7f7fa17a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:02.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.136+0000 7f7fa17a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:02.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.136+0000 7f7fa17a5640 10 monclient(hunting): picked mon.noname-a con 0x7f7f9c0a9ac0 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:02.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.136+0000 7f7fa17a5640 10 monclient(hunting): picked mon.noname-c con 0x7f7f9c0af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:02.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.136+0000 7f7fa17a5640 10 monclient(hunting): picked mon.noname-b con 0x7f7f9c072470 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:02.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.136+0000 7f7fa17a5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9c1e7520 con 0x7f7f9c0a9ac0 2024-03-29T11:43:02.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.136+0000 7f7fa17a5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9c1412e0 con 0x7f7f9c072470 2024-03-29T11:43:02.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.136+0000 7f7fa17a5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9c13c8c0 con 0x7f7f9c0af900 2024-03-29T11:43:02.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.136+0000 7f7fa17a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:02.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.136+0000 7f7fa17a5640 10 monclient(hunting): authenticate will time out at 1364.263788s 2024-03-29T11:43:02.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.142+0000 7f7f9a7fc640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f7f9c0af900 0x7f7f9c071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51951/0 (socket says 172.21.15.155:51951) 2024-03-29T11:43:02.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.142+0000 7f7f9a7fc640 1 -- 172.21.15.155:0/1916562371 learned_addr learned my addr 172.21.15.155:0/1916562371 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:02.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.142+0000 7f7f837fe640 10 client.?.objecter ms_handle_connect 0x7f7f9c0af900 2024-03-29T11:43:02.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.142+0000 7f7f837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:02.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.142+0000 7f7f837fe640 10 client.?.objecter ms_handle_connect 0x7f7f9c0a9ac0 2024-03-29T11:43:02.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.142+0000 7f7f837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:02.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 client.?.objecter ms_handle_connect 0x7f7f9c072470 2024-03-29T11:43:02.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:02.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013206402 0 0) 0x7f7f9c1e7520 con 0x7f7f9c0a9ac0 2024-03-29T11:43:02.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:02.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:02.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:02.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:43:02.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f88004eb0 con 0x7f7f9c0a9ac0 2024-03-29T11:43:02.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433678977 0 0) 0x7f7f9c13c8c0 con 0x7f7f9c0af900 2024-03-29T11:43:02.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:02.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:02.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:02.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:43:02.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f9c1e7520 con 0x7f7f9c0af900 2024-03-29T11:43:02.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103501300 0 0) 0x7f7f9c1412e0 con 0x7f7f9c072470 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f9c13c8c0 con 0x7f7f9c072470 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3433887560 0 0) 0x7f7f88004eb0 con 0x7f7f9c0a9ac0 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f9c1412e0 con 0x7f7f9c0a9ac0 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1171307341 0 0) 0x7f7f9c13c8c0 con 0x7f7f9c072470 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f88004eb0 con 0x7f7f9c072470 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7f900018b0 con 0x7f7f9c0a9ac0 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): dump: 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:02.156 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7f8c001660 con 0x7f7f9c072470 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.146+0000 7f7f837fe640 10 monclient(hunting): dump: 2024-03-29T11:43:02.157 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 47042541 0 0) 0x7f7f9c1e7520 con 0x7f7f9c0af900 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f9c13c8c0 con 0x7f7f9c0af900 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1398863223 0 0) 0x7f7f88004eb0 con 0x7f7f9c072470 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 >> v1:172.21.15.155:6790/0 conn(0x7f7f9c0af900 legacy=0x7f7f9c071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 >> v1:172.21.15.155:6789/0 conn(0x7f7f9c0a9ac0 legacy=0x7f7f9c140bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:02.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 1 monclient: found mon.b 2024-03-29T11:43:02.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:02.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:02.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c13b350 con 0x7f7f9c072470 2024-03-29T11:43:02.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:02.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:02.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:02.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f8c001b70 con 0x7f7f9c072470 2024-03-29T11:43:02.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:02.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f7f8c001d60 con 0x7f7f9c072470 2024-03-29T11:43:02.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:02.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:02.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.147+0000 7f7f837fe640 10 monclient: dump: 2024-03-29T11:43:02.160 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:02.161 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:02.161 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:02.161 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:02.161 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:02.162 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:02.162 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:02.162 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:02.162 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:02.162 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:02.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.148+0000 7f7f98ff9640 4 set_mon_vals no callback set 2024-03-29T11:43:02.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.148+0000 7f7fa17a5640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:43:02.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.148+0000 7f7fa17a5640 10 monclient: _renew_subs 2024-03-29T11:43:02.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.148+0000 7f7fa17a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:02.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.148+0000 7f7fa17a5640 1 -- 172.21.15.155:0/1916562371 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f9c1e8110 con 0x7f7f9c072470 2024-03-29T11:43:02.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.148+0000 7f7fa17a5640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.148+0000 7f7fa17a5640 10 monclient: _renew_subs 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.148+0000 7f7fa17a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.148+0000 7f7fa17a5640 1 -- 172.21.15.155:0/1916562371 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f9c140080 con 0x7f7f9c072470 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.148+0000 7f7fa17a5640 1 librados: init done 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.152+0000 7f7fa17a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.152+0000 7f7fa17a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.152+0000 7f7fa17a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.152+0000 7f7fa17a5640 1 -- 172.21.15.155:0/1916562371 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f9c13e600 con 0x7f7f9c072470 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.152+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f7f8c01a7f0 con 0x7f7f9c072470 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.153+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f7f8c0846f0 con 0x7f7f9c072470 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.153+0000 7f7f837fe640 10 client.4483.objecter ms_dispatch 0x7f7f9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.153+0000 7f7f837fe640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.153+0000 7f7f837fe640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:02.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.153+0000 7f7f837fe640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-29T11:43:02.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.153+0000 7f7f9affd640 10 monclient: get_auth_request con 0x7f7f8806e180 auth_method 0 2024-03-29T11:43:02.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.154+0000 7f7f837fe640 10 client.4483.objecter ms_handle_connect 0x7f7f8806e180 2024-03-29T11:43:02.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.160+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f8c04ffd0 con 0x7f7f9c072470 2024-03-29T11:43:02.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.160+0000 7f7f837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:02.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.160+0000 7f7f837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:02.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.496+0000 7f7fa17a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:02.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.496+0000 7f7fa17a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:02.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.496+0000 7f7fa17a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:02.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.496+0000 7f7fa17a5640 1 -- 172.21.15.155:0/1916562371 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f9c13b4e0 con 0x7f7f9c072470 2024-03-29T11:43:02.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.498+0000 7f7f837fe640 1 -- 172.21.15.155:0/1916562371 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f7f9c13b4e0 con 0x7f7f9c072470 2024-03-29T11:43:02.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.498+0000 7f7f837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:02.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.498+0000 7f7f837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:02.502 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:43:02.503 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:02.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.503+0000 7f7fa17a5640 10 librados: watch_flush enter 2024-03-29T11:43:02.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.503+0000 7f7fa17a5640 10 librados: watch_flush exit 2024-03-29T11:43:02.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.503+0000 7f7fa17a5640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-29T11:43:02.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.503+0000 7f7fa17a5640 10 client.4483.objecter successfully canceled tick 2024-03-29T11:43:02.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.503+0000 7f7fa17a5640 1 -- 172.21.15.155:0/1916562371 >> v1:172.21.15.155:6816/1760814708 conn(0x7f7f8806e180 legacy=0x7f7f88070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.503+0000 7f7fa17a5640 10 monclient: shutdown 2024-03-29T11:43:02.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.503+0000 7f7fa17a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:02.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.503+0000 7f7fa17a5640 1 -- 172.21.15.155:0/1916562371 >> v1:172.21.15.183:6789/0 conn(0x7f7f9c072470 legacy=0x7f7f9c1e7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.503+0000 7f7fa17a5640 1 -- 172.21.15.155:0/1916562371 shutdown_connections 2024-03-29T11:43:02.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.504+0000 7f7fa17a5640 1 -- 172.21.15.155:0/1916562371 >> 172.21.15.155:0/1916562371 conn(0x7f7f9c0b35e0 msgr2=0x7f7f9c0acff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:02.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.504+0000 7f7fa17a5640 1 -- 172.21.15.155:0/1916562371 shutdown_connections 2024-03-29T11:43:02.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.504+0000 7f7fa17a5640 1 -- 172.21.15.155:0/1916562371 wait complete. 2024-03-29T11:43:02.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:02.504+0000 7f7fa17a5640 1 librados: shutdown 2024-03-29T11:43:02.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:09.333 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.474+0000 7fa17f233640 10 monclient: get_monmap_and_config 2024-03-29T11:43:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.474+0000 7fa17f233640 10 monclient: build_initial_monmap 2024-03-29T11:43:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.474+0000 7fa17f233640 10 monclient: monmap: 2024-03-29T11:43:09.478 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:09.478 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:09.478 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:43:09.475538+0000 2024-03-29T11:43:09.478 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:43:09.475538+0000 2024-03-29T11:43:09.478 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:09.479 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:09.479 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:09.479 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:43:09.479 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:43:09.479 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:09.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.476+0000 7fa17f233640 1 Processor -- start 2024-03-29T11:43:09.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.478+0000 7fa17f233640 1 -- start start 2024-03-29T11:43:09.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.478+0000 7fa17f233640 10 monclient: init 2024-03-29T11:43:09.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.479+0000 7fa17f233640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:09.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.479+0000 7fa17f233640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:09.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.479+0000 7fa17f233640 10 monclient(hunting): picked mon.noname-a con 0x7fa1781457f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:09.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.479+0000 7fa17f233640 10 monclient(hunting): picked mon.noname-b con 0x7fa17813f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:09.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.479+0000 7fa17f233640 10 monclient(hunting): picked mon.noname-c con 0x7fa1780a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:09.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.479+0000 7fa17f233640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1780aaef0 con 0x7fa1781457f0 2024-03-29T11:43:09.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.479+0000 7fa17f233640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa178145bd0 con 0x7fa1780a9d70 2024-03-29T11:43:09.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.479+0000 7fa17f233640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa178145eb0 con 0x7fa17813f650 2024-03-29T11:43:09.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.479+0000 7fa17f233640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:09.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.479+0000 7fa17f233640 10 monclient(hunting): authenticate will time out at 1371.606681s 2024-03-29T11:43:09.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.480+0000 7fa177fff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa17813f650 0x7fa1780b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:53420/0 (socket says 172.21.15.155:53420) 2024-03-29T11:43:09.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.480+0000 7fa177fff640 1 -- 172.21.15.155:0/1375580257 learned_addr learned my addr 172.21.15.155:0/1375580257 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:09.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.481+0000 7fa1777fe640 1 -- 172.21.15.155:0/1375580257 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517346473 0 0) 0x7fa178145eb0 con 0x7fa17813f650 2024-03-29T11:43:09.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.481+0000 7fa1777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:09.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.481+0000 7fa1777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:09.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.481+0000 7fa1777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:09.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.481+0000 7fa1777fe640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:43:09.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.481+0000 7fa1777fe640 1 -- 172.21.15.155:0/1375580257 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa168003520 con 0x7fa17813f650 2024-03-29T11:43:09.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.482+0000 7fa1777fe640 1 -- 172.21.15.155:0/1375580257 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 216457052 0 0) 0x7fa168003520 con 0x7fa17813f650 2024-03-29T11:43:09.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.482+0000 7fa1777fe640 1 -- 172.21.15.155:0/1375580257 >> v1:172.21.15.183:6789/0 conn(0x7fa1780a9d70 legacy=0x7fa17814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.482+0000 7fa1777fe640 1 -- 172.21.15.155:0/1375580257 >> v1:172.21.15.155:6789/0 conn(0x7fa1781457f0 legacy=0x7fa17813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.482+0000 7fa1777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 1 monclient: found mon.noname-b 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 1 -- 172.21.15.155:0/1375580257 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa178146340 con 0x7fa17813f650 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa17f233640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa17f233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:09.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 1 -- 172.21.15.155:0/1375580257 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa16c001690 con 0x7fa17813f650 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 10 monclient: dump: 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:09.490 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:09.491 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:09.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa17f233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:09.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 1 -- 172.21.15.155:0/1375580257 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa16c001860 con 0x7fa17813f650 2024-03-29T11:43:09.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa1777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:09.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa17f233640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:09.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.483+0000 7fa17f233640 4 set_mon_vals no callback set 2024-03-29T11:43:09.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.484+0000 7fa17f233640 10 monclient: shutdown 2024-03-29T11:43:09.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.484+0000 7fa17f233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:09.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.484+0000 7fa17f233640 1 -- 172.21.15.155:0/1375580257 >> v1:172.21.15.155:6790/0 conn(0x7fa17813f650 legacy=0x7fa1780b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:09.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.485+0000 7fa17f233640 1 -- 172.21.15.155:0/1375580257 shutdown_connections 2024-03-29T11:43:09.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.485+0000 7fa17f233640 1 -- 172.21.15.155:0/1375580257 >> 172.21.15.155:0/1375580257 conn(0x7fa178066a00 msgr2=0x7fa178066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:09.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.485+0000 7fa17f233640 1 -- 172.21.15.155:0/1375580257 shutdown_connections 2024-03-29T11:43:09.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.485+0000 7fa17f233640 1 -- 172.21.15.155:0/1375580257 wait complete. 2024-03-29T11:43:09.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.486+0000 7fa17f233640 10 monclient: build_initial_monmap 2024-03-29T11:43:09.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.486+0000 7fa17f233640 10 monclient: monmap: 2024-03-29T11:43:09.494 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:09.494 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:09.494 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:43:09.494 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:43:09.494 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:09.495 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:09.495 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:09.495 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:43:09.495 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:43:09.495 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:09.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.486+0000 7fa17f233640 1 librados: starting msgr at 2024-03-29T11:43:09.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.486+0000 7fa17f233640 1 librados: starting objecter 2024-03-29T11:43:09.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.486+0000 7fa17f233640 1 Processor -- start 2024-03-29T11:43:09.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.487+0000 7fa17f233640 1 -- start start 2024-03-29T11:43:09.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.487+0000 7fa17f233640 1 librados: setting wanted keys 2024-03-29T11:43:09.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.487+0000 7fa17f233640 1 librados: calling monclient init 2024-03-29T11:43:09.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.487+0000 7fa17f233640 10 monclient: init 2024-03-29T11:43:09.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.488+0000 7fa17f233640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:09.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.488+0000 7fa17f233640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:09.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.491+0000 7fa17f233640 10 monclient(hunting): picked mon.noname-a con 0x7fa1780a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:09.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.491+0000 7fa17cfcf640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fa1780a9d70 0x7fa1781e3b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:59567/0 (socket says 172.21.15.155:59567) 2024-03-29T11:43:09.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.491+0000 7fa17cfcf640 1 -- 172.21.15.155:0/3831971898 learned_addr learned my addr 172.21.15.155:0/3831971898 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:09.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.491+0000 7fa17f233640 10 monclient(hunting): picked mon.noname-c con 0x7fa17813f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:09.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.491+0000 7fa17f233640 10 monclient(hunting): picked mon.noname-b con 0x7fa1781457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:09.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 client.?.objecter ms_handle_connect 0x7fa1780a9d70 2024-03-29T11:43:09.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:09.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1781d8120 con 0x7fa1780a9d70 2024-03-29T11:43:09.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1781412c0 con 0x7fa1781457f0 2024-03-29T11:43:09.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa178144ed0 con 0x7fa17813f650 2024-03-29T11:43:09.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa17f233640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:09.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa17f233640 10 monclient(hunting): authenticate will time out at 1371.619000s 2024-03-29T11:43:09.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778455197 0 0) 0x7fa1781d8120 con 0x7fa1780a9d70 2024-03-29T11:43:09.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:09.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:09.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:09.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:43:09.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa15c004eb0 con 0x7fa1780a9d70 2024-03-29T11:43:09.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 client.?.objecter ms_handle_connect 0x7fa17813f650 2024-03-29T11:43:09.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2895632950 0 0) 0x7fa15c004eb0 con 0x7fa1780a9d70 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1781d8120 con 0x7fa1780a9d70 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa1640017d0 con 0x7fa1780a9d70 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 monclient(hunting): dump: 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:09.500 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:09.501 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:09.501 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:09.501 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:09.501 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:09.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:09.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.492+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1552240823 0 0) 0x7fa1781d8120 con 0x7fa1780a9d70 2024-03-29T11:43:09.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.493+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 >> v1:172.21.15.155:6790/0 conn(0x7fa17813f650 legacy=0x7fa1781d7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:09.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.493+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 >> v1:172.21.15.183:6789/0 conn(0x7fa1781457f0 legacy=0x7fa1781d7a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:09.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa1757fa640 10 monclient: _finish_hunting 0 2024-03-29T11:43:09.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa1757fa640 1 monclient: found mon.a 2024-03-29T11:43:09.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa1757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:09.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa1757fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:43:09.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa178142ce0 con 0x7fa1780a9d70 2024-03-29T11:43:09.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa1757fa640 10 monclient: _finish_auth 0 2024-03-29T11:43:09.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa1757fa640 10 monclient: _check_auth_tickets 2024-03-29T11:43:09.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa1757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:09.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa17f233640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:43:09.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa17f233640 10 monclient: _renew_subs 2024-03-29T11:43:09.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa17f233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:43:09.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.495+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17813dd90 con 0x7fa1780a9d70 2024-03-29T11:43:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.496+0000 7fa17f233640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.496+0000 7fa17f233640 10 monclient: _renew_subs 2024-03-29T11:43:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.496+0000 7fa17f233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:43:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.496+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17813ff80 con 0x7fa1780a9d70 2024-03-29T11:43:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.496+0000 7fa17f233640 1 librados: init done 2024-03-29T11:43:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.497+0000 7fa17f233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.497+0000 7fa17f233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.497+0000 7fa17f233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:43:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.498+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17813e130 con 0x7fa1780a9d70 2024-03-29T11:43:09.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.498+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa164001a60 con 0x7fa1780a9d70 2024-03-29T11:43:09.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.498+0000 7fa1757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:09.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.498+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fa164004a80 con 0x7fa1780a9d70 2024-03-29T11:43:09.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.498+0000 7fa1757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:09.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.498+0000 7fa1757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:09.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.498+0000 7fa1757fa640 10 monclient: dump: 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.498+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fa164004c60 con 0x7fa1780a9d70 2024-03-29T11:43:09.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.499+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fa164084be0 con 0x7fa1780a9d70 2024-03-29T11:43:09.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.499+0000 7fa1757fa640 10 client.4524.objecter ms_dispatch 0x7fa178000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:09.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.499+0000 7fa1757fa640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:09.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.499+0000 7fa1757fa640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:09.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.499+0000 7fa1757fa640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-29T11:43:09.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.499+0000 7fa1767fc640 4 set_mon_vals no callback set 2024-03-29T11:43:09.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.501+0000 7fa177fff640 10 monclient: get_auth_request con 0x7fa15c06dd10 auth_method 0 2024-03-29T11:43:09.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.502+0000 7fa1757fa640 10 client.4524.objecter ms_handle_connect 0x7fa15c06dd10 2024-03-29T11:43:09.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.507+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa164050370 con 0x7fa1780a9d70 2024-03-29T11:43:09.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.507+0000 7fa1757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:09.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.507+0000 7fa1757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:09.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.835+0000 7fa17f233640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:09.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.835+0000 7fa17f233640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:09.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.835+0000 7fa17f233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:43:09.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.835+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa178145060 con 0x7fa1780a9d70 2024-03-29T11:43:09.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.835+0000 7fa1757fa640 1 -- 172.21.15.155:0/3831971898 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fa164052470 con 0x7fa1780a9d70 2024-03-29T11:43:09.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.835+0000 7fa1757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:09.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.835+0000 7fa1757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:09.840 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:43:09.841 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:09.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.841+0000 7fa17f233640 10 librados: watch_flush enter 2024-03-29T11:43:09.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.841+0000 7fa17f233640 10 librados: watch_flush exit 2024-03-29T11:43:09.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.842+0000 7fa17f233640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-29T11:43:09.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.842+0000 7fa17f233640 10 client.4524.objecter successfully canceled tick 2024-03-29T11:43:09.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.842+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 >> v1:172.21.15.155:6816/1760814708 conn(0x7fa15c06dd10 legacy=0x7fa15c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:09.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.842+0000 7fa17f233640 10 monclient: shutdown 2024-03-29T11:43:09.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.842+0000 7fa17f233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:09.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.842+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 >> v1:172.21.15.155:6789/0 conn(0x7fa1780a9d70 legacy=0x7fa1781e3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:09.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.842+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 shutdown_connections 2024-03-29T11:43:09.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.842+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 >> 172.21.15.155:0/3831971898 conn(0x7fa178066a00 msgr2=0x7fa1780b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:09.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.842+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 shutdown_connections 2024-03-29T11:43:09.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.842+0000 7fa17f233640 1 -- 172.21.15.155:0/3831971898 wait complete. 2024-03-29T11:43:09.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:09.842+0000 7fa17f233640 1 librados: shutdown 2024-03-29T11:43:09.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:16.670 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:16.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.822+0000 7f06a438f640 10 monclient: get_monmap_and_config 2024-03-29T11:43:16.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.822+0000 7f06a438f640 10 monclient: build_initial_monmap 2024-03-29T11:43:16.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.822+0000 7f06a438f640 10 monclient: monmap: 2024-03-29T11:43:16.826 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:16.826 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:16.826 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:43:16.823161+0000 2024-03-29T11:43:16.826 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:43:16.823161+0000 2024-03-29T11:43:16.826 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:16.826 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:16.826 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:16.826 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:43:16.826 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:43:16.826 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:16.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.823+0000 7f06a438f640 1 Processor -- start 2024-03-29T11:43:16.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.823+0000 7f06a438f640 1 -- start start 2024-03-29T11:43:16.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.823+0000 7f06a438f640 10 monclient: init 2024-03-29T11:43:16.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a438f640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a438f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a438f640 10 monclient(hunting): picked mon.noname-c con 0x7f069c1457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a438f640 10 monclient(hunting): picked mon.noname-b con 0x7f069c13f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a438f640 10 monclient(hunting): picked mon.noname-a con 0x7f069c0a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a438f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0aaef0 con 0x7f069c0a9d70 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a438f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c145bd0 con 0x7f069c1457f0 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a438f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c145eb0 con 0x7f069c13f650 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a438f640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a438f640 10 monclient(hunting): authenticate will time out at 1378.951330s 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a192a640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f069c13f650 0x7f069c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:35686/0 (socket says 172.21.15.155:35686) 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a292c640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f069c0a9d70 0x7f069c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:56608/0 (socket says 172.21.15.155:56608) 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a192a640 1 -- 172.21.15.155:0/2862587541 learned_addr learned my addr 172.21.15.155:0/2862587541 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.825+0000 7f06a1129640 1 -- 172.21.15.155:0/2862587541 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257353730 0 0) 0x7f069c0aaef0 con 0x7f069c0a9d70 2024-03-29T11:43:16.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient(hunting): _negotiate 2024-03-29T11:43:16.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:16.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:16.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:43:16.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 1 -- 172.21.15.155:0/2862587541 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0688003520 con 0x7f069c0a9d70 2024-03-29T11:43:16.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 1 -- 172.21.15.155:0/2862587541 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17586146 0 0) 0x7f069c145eb0 con 0x7f069c13f650 2024-03-29T11:43:16.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient(hunting): _negotiate 2024-03-29T11:43:16.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:16.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:16.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:43:16.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 1 -- 172.21.15.155:0/2862587541 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c0aaef0 con 0x7f069c13f650 2024-03-29T11:43:16.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 1 -- 172.21.15.155:0/2862587541 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 871620353 0 0) 0x7f069c0aaef0 con 0x7f069c13f650 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 1 -- 172.21.15.155:0/2862587541 >> v1:172.21.15.183:6789/0 conn(0x7f069c1457f0 legacy=0x7f069c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 1 -- 172.21.15.155:0/2862587541 >> v1:172.21.15.155:6789/0 conn(0x7f069c0a9d70 legacy=0x7f069c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient: _finish_hunting 0 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 1 monclient: found mon.noname-b 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 1 -- 172.21.15.155:0/2862587541 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c146340 con 0x7f069c13f650 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient: _finish_auth 0 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient: _check_auth_tickets 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 1 -- 172.21.15.155:0/2862587541 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f068c001670 con 0x7f069c13f650 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:16.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a1129640 10 monclient: dump: 2024-03-29T11:43:16.833 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:16.833 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:16.833 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:16.833 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:16.833 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a438f640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.826+0000 7f06a438f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.827+0000 7f06a1129640 1 -- 172.21.15.155:0/2862587541 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f068c001890 con 0x7f069c13f650 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.827+0000 7f06a1129640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.827+0000 7f06a1129640 1 -- 172.21.15.155:0/2862587541 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f068c001c20 con 0x7f069c13f650 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.827+0000 7f06a1129640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.827+0000 7f06a1129640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:16.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.827+0000 7f06a1129640 10 monclient: dump: 2024-03-29T11:43:16.835 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:16.835 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:16.835 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:16.835 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:16.835 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:16.835 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:16.835 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:16.835 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:16.835 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:16.835 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.827+0000 7f06a438f640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.827+0000 7f06a438f640 4 set_mon_vals no callback set 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.828+0000 7f06a438f640 10 monclient: shutdown 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.828+0000 7f06a438f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.828+0000 7f06a438f640 1 -- 172.21.15.155:0/2862587541 >> v1:172.21.15.155:6790/0 conn(0x7f069c13f650 legacy=0x7f069c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.828+0000 7f06a438f640 1 -- 172.21.15.155:0/2862587541 shutdown_connections 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.828+0000 7f06a438f640 1 -- 172.21.15.155:0/2862587541 >> 172.21.15.155:0/2862587541 conn(0x7f069c066a00 msgr2=0x7f069c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.828+0000 7f06a438f640 1 -- 172.21.15.155:0/2862587541 shutdown_connections 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.828+0000 7f06a438f640 1 -- 172.21.15.155:0/2862587541 wait complete. 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.829+0000 7f06a438f640 10 monclient: build_initial_monmap 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.829+0000 7f06a438f640 10 monclient: monmap: 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:43:16.836 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:43:16.837 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:16.837 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:16.837 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:16.837 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:43:16.837 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:43:16.837 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:16.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.829+0000 7f06a438f640 1 librados: starting msgr at 2024-03-29T11:43:16.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.829+0000 7f06a438f640 1 librados: starting objecter 2024-03-29T11:43:16.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.829+0000 7f06a438f640 1 Processor -- start 2024-03-29T11:43:16.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.829+0000 7f06a438f640 1 -- start start 2024-03-29T11:43:16.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.829+0000 7f06a438f640 1 librados: setting wanted keys 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.829+0000 7f06a438f640 1 librados: calling monclient init 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.829+0000 7f06a438f640 10 monclient: init 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.830+0000 7f06a438f640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.830+0000 7f06a438f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.830+0000 7f06a438f640 10 monclient(hunting): picked mon.noname-c con 0x7f069c0a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.830+0000 7f06a438f640 10 monclient(hunting): picked mon.noname-a con 0x7f069c13f650 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.830+0000 7f06a212b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f069c0a9d70 0x7f069c1401f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36136/0 (socket says 172.21.15.155:36136) 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f06a438f640 10 monclient(hunting): picked mon.noname-b con 0x7f069c1457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f06a212b640 1 -- 172.21.15.155:0/3544031742 learned_addr learned my addr 172.21.15.155:0/3544031742 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c137890 con 0x7f069c13f650 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c141c20 con 0x7f069c1457f0 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0ac580 con 0x7f069c0a9d70 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f06a438f640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f06a438f640 10 monclient(hunting): authenticate will time out at 1378.957039s 2024-03-29T11:43:16.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 client.?.objecter ms_handle_connect 0x7f069c0a9d70 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 client.?.objecter ms_handle_connect 0x7f069c13f650 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656149890 0 0) 0x7f069c0ac580 con 0x7f069c0a9d70 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0680004eb0 con 0x7f069c0a9d70 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514474910 0 0) 0x7f069c137890 con 0x7f069c13f650 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:16.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c0ac580 con 0x7f069c13f650 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4190342897 0 0) 0x7f0680004eb0 con 0x7f069c0a9d70 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.831+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f069c137890 con 0x7f069c0a9d70 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f0698003fe0 con 0x7f069c0a9d70 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 10 monclient(hunting): dump: 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:16.847 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 10 client.?.objecter ms_handle_connect 0x7f069c1457f0 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143664792 0 0) 0x7f069c141c20 con 0x7f069c1457f0 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.832+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0680004eb0 con 0x7f069c1457f0 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1398527689 0 0) 0x7f069c137890 con 0x7f069c0a9d70 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 >> v1:172.21.15.183:6789/0 conn(0x7f069c1457f0 legacy=0x7f069c137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:16.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 >> v1:172.21.15.155:6789/0 conn(0x7f069c13f650 legacy=0x7f069c136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f0692ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f0692ffd640 1 monclient: found mon.c 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f0692ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f0692ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c0ac860 con 0x7f069c0a9d70 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f0692ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f0692ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f0692ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f06a438f640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f06a438f640 10 monclient: _renew_subs 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f06a438f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c137c10 con 0x7f069c0a9d70 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f06a438f640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f06a438f640 10 monclient: _renew_subs 2024-03-29T11:43:16.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f06a438f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c0ab800 con 0x7f069c0a9d70 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.833+0000 7f06a438f640 1 librados: init done 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.834+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0698004140 con 0x7f069c0a9d70 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.834+0000 7f0692ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.834+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f06980010f0 con 0x7f069c0a9d70 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.834+0000 7f0692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.834+0000 7f0692ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.834+0000 7f0692ffd640 10 monclient: dump: 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:16.850 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.834+0000 7f06a0928640 4 set_mon_vals no callback set 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.835+0000 7f06a438f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.835+0000 7f06a438f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.835+0000 7f06a438f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.835+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0684005080 con 0x7f069c0a9d70 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.836+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f0698001250 con 0x7f069c0a9d70 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.837+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f0698084a00 con 0x7f069c0a9d70 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.837+0000 7f0692ffd640 10 client.4589.objecter ms_dispatch 0x7f069c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.837+0000 7f0692ffd640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.837+0000 7f0692ffd640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:16.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.837+0000 7f0692ffd640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:43:16.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.837+0000 7f06a192a640 10 monclient: get_auth_request con 0x7f068006dbc0 auth_method 0 2024-03-29T11:43:16.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.837+0000 7f0692ffd640 10 client.4589.objecter ms_handle_connect 0x7f068006dbc0 2024-03-29T11:43:16.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.845+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06980502e0 con 0x7f069c0a9d70 2024-03-29T11:43:16.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.845+0000 7f0692ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:16.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:16.845+0000 7f0692ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:17.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.237+0000 7f06a438f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:17.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.237+0000 7f06a438f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:17.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.237+0000 7f06a438f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:17.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.237+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0684005330 con 0x7f069c0a9d70 2024-03-29T11:43:17.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.238+0000 7f0692ffd640 1 -- 172.21.15.155:0/3544031742 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f0698052210 con 0x7f069c0a9d70 2024-03-29T11:43:17.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.238+0000 7f0692ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:17.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.238+0000 7f0692ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:17.241 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:43:17.241 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:17.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 10 librados: watch_flush enter 2024-03-29T11:43:17.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 10 librados: watch_flush exit 2024-03-29T11:43:17.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:43:17.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:43:17.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 >> v1:172.21.15.155:6816/1760814708 conn(0x7f068006dbc0 legacy=0x7f0680070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:17.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 10 monclient: shutdown 2024-03-29T11:43:17.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:17.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 >> v1:172.21.15.155:6790/0 conn(0x7f069c0a9d70 legacy=0x7f069c1401f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:17.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 shutdown_connections 2024-03-29T11:43:17.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 >> 172.21.15.155:0/3544031742 conn(0x7f069c066a00 msgr2=0x7f069c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:17.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 shutdown_connections 2024-03-29T11:43:17.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 1 -- 172.21.15.155:0/3544031742 wait complete. 2024-03-29T11:43:17.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:17.242+0000 7f06a438f640 1 librados: shutdown 2024-03-29T11:43:17.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:24.068 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:24.224 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.221+0000 7fafe5cf2640 10 monclient: get_monmap_and_config 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.221+0000 7fafe5cf2640 10 monclient: build_initial_monmap 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.222+0000 7fafe5cf2640 10 monclient: monmap: 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:43:24.222423+0000 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:43:24.222423+0000 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:43:24.225 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:24.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.223+0000 7fafe5cf2640 1 Processor -- start 2024-03-29T11:43:24.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.223+0000 7fafe5cf2640 1 -- start start 2024-03-29T11:43:24.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.223+0000 7fafe5cf2640 10 monclient: init 2024-03-29T11:43:24.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.224+0000 7fafe5cf2640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:24.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.224+0000 7fafe5cf2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:24.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.224+0000 7fafe5cf2640 10 monclient(hunting): picked mon.noname-a con 0x7fafd8006eb0 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:24.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.224+0000 7fafe5cf2640 10 monclient(hunting): picked mon.noname-c con 0x7fafd8038f10 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:24.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.224+0000 7fafe5cf2640 10 monclient(hunting): picked mon.noname-b con 0x7fafd8036690 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:24.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.224+0000 7fafe5cf2640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd8033c90 con 0x7fafd8006eb0 2024-03-29T11:43:24.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.224+0000 7fafe5cf2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd80389f0 con 0x7fafd8038f10 2024-03-29T11:43:24.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.224+0000 7fafe5cf2640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd8030890 con 0x7fafd8036690 2024-03-29T11:43:24.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.224+0000 7fafe5cf2640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:24.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.224+0000 7fafe5cf2640 10 monclient(hunting): authenticate will time out at 1386.350075s 2024-03-29T11:43:24.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.225+0000 7fafe4cf0640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fafd8006eb0 0x7fafd8031760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34728/0 (socket says 172.21.15.155:34728) 2024-03-29T11:43:24.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.225+0000 7fafe4cf0640 1 -- 172.21.15.155:0/2620891824 learned_addr learned my addr 172.21.15.155:0/2620891824 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:24.230 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.227+0000 7fafdf7fe640 1 -- 172.21.15.155:0/2620891824 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416593559 0 0) 0x7fafd8033c90 con 0x7fafd8006eb0 2024-03-29T11:43:24.230 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.227+0000 7fafdf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:24.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.227+0000 7fafdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:24.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.227+0000 7fafdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:24.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.227+0000 7fafdf7fe640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:43:24.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.227+0000 7fafdf7fe640 1 -- 172.21.15.155:0/2620891824 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc8003520 con 0x7fafd8006eb0 2024-03-29T11:43:24.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.228+0000 7fafdf7fe640 1 -- 172.21.15.155:0/2620891824 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 390604838 0 0) 0x7fafc8003520 con 0x7fafd8006eb0 2024-03-29T11:43:24.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.228+0000 7fafdf7fe640 1 -- 172.21.15.155:0/2620891824 >> v1:172.21.15.155:6790/0 conn(0x7fafd8036690 legacy=0x7fafd8036b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:24.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdffff640 1 -- 172.21.15.155:0/2620891824 >> v1:172.21.15.183:6789/0 conn(0x7fafd8038f10 legacy=0x7fafd8039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:43:24.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdffff640 1 -- 172.21.15.155:0/2620891824 >> v1:172.21.15.183:6789/0 conn(0x7fafd8038f10 legacy=0x7fafd8039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:43:24.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdffff640 1 --1- 172.21.15.155:0/2620891824 >> v1:172.21.15.183:6789/0 conn(0x7fafd8038f10 0x7fafd8039330 :-1 s=OPENED pgs=1078366 cs=1 l=1).handle_message read tag failed 2024-03-29T11:43:24.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdffff640 1 --1- 172.21.15.155:0/2620891824 >> v1:172.21.15.183:6789/0 conn(0x7fafd8038f10 0x7fafd8039330 :-1 s=OPENED pgs=1078366 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:43:24.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 1 -- 172.21.15.155:0/2620891824 >> v1:172.21.15.183:6789/0 conn(0x7fafd8038f10 legacy=0x7fafd8039330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:43:24.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:24.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 1 monclient: found mon.noname-a 2024-03-29T11:43:24.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:24.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:43:24.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 1 -- 172.21.15.155:0/2620891824 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd80cbc40 con 0x7fafd8006eb0 2024-03-29T11:43:24.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:24.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:24.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:24.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: ms_handle_reset stray mon v1:172.21.15.183:6789/0 2024-03-29T11:43:24.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 1 -- 172.21.15.155:0/2620891824 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fafd0001690 con 0x7fafd8006eb0 2024-03-29T11:43:24.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:24.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:24.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: dump: 2024-03-29T11:43:24.234 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:24.234 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:24.234 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:24.235 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:24.235 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:24.235 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:24.235 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:24.235 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:24.235 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:24.235 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:24.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 1 -- 172.21.15.155:0/2620891824 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafd00018f0 con 0x7fafd8006eb0 2024-03-29T11:43:24.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:24.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 1 -- 172.21.15.155:0/2620891824 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fafd0001d40 con 0x7fafd8006eb0 2024-03-29T11:43:24.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:24.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:24.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafdf7fe640 10 monclient: dump: 2024-03-29T11:43:24.236 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:24.236 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:24.236 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:24.236 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:24.236 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:24.237 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:24.237 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:24.237 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:24.237 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:24.237 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:24.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafe5cf2640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:43:24.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafe5cf2640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:24.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.229+0000 7fafe5cf2640 4 set_mon_vals no callback set 2024-03-29T11:43:24.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.230+0000 7fafe5cf2640 10 monclient: shutdown 2024-03-29T11:43:24.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.230+0000 7fafe5cf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:24.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.230+0000 7fafe5cf2640 1 -- 172.21.15.155:0/2620891824 >> v1:172.21.15.155:6789/0 conn(0x7fafd8006eb0 legacy=0x7fafd8031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:24.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.230+0000 7fafe5cf2640 1 -- 172.21.15.155:0/2620891824 shutdown_connections 2024-03-29T11:43:24.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.230+0000 7fafe5cf2640 1 -- 172.21.15.155:0/2620891824 >> 172.21.15.155:0/2620891824 conn(0x7fafd8005c20 msgr2=0x7fafd8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:24.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.230+0000 7fafe5cf2640 1 -- 172.21.15.155:0/2620891824 shutdown_connections 2024-03-29T11:43:24.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.231+0000 7fafe5cf2640 1 -- 172.21.15.155:0/2620891824 wait complete. 2024-03-29T11:43:24.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.231+0000 7fafe5cf2640 10 monclient: build_initial_monmap 2024-03-29T11:43:24.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.231+0000 7fafe5cf2640 10 monclient: monmap: 2024-03-29T11:43:24.239 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:24.239 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.232+0000 7fafe5cf2640 1 librados: starting msgr at 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.232+0000 7fafe5cf2640 1 librados: starting objecter 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.232+0000 7fafe5cf2640 1 Processor -- start 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.232+0000 7fafe5cf2640 1 -- start start 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.232+0000 7fafe5cf2640 1 librados: setting wanted keys 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.232+0000 7fafe5cf2640 1 librados: calling monclient init 2024-03-29T11:43:24.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.232+0000 7fafe5cf2640 10 monclient: init 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.233+0000 7fafe5cf2640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.233+0000 7fafe5cf2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.233+0000 7fafe5cf2640 10 monclient(hunting): picked mon.noname-c con 0x7fafd8036690 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.233+0000 7fafe5cf2640 10 monclient(hunting): picked mon.noname-b con 0x7fafd8038f10 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.233+0000 7fafe5cf2640 10 monclient(hunting): picked mon.noname-a con 0x7fafd80c91d0 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.233+0000 7fafe5cf2640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd800b2f0 con 0x7fafd80c91d0 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.233+0000 7fafe5cf2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd8030150 con 0x7fafd8038f10 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.233+0000 7fafe5cf2640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd80c1b20 con 0x7fafd8036690 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.233+0000 7fafe5cf2640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.233+0000 7fafe5cf2640 10 monclient(hunting): authenticate will time out at 1386.359659s 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.234+0000 7fafe54f1640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fafd80c91d0 0x7fafd80097b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:39476/0 (socket says 172.21.15.155:39476) 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.234+0000 7fafe54f1640 1 -- 172.21.15.155:0/3292895183 learned_addr learned my addr 172.21.15.155:0/3292895183 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.235+0000 7fafdd7fa640 10 client.?.objecter ms_handle_connect 0x7fafd80c91d0 2024-03-29T11:43:24.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.235+0000 7fafdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:24.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 client.?.objecter ms_handle_connect 0x7fafd8038f10 2024-03-29T11:43:24.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:24.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 client.?.objecter ms_handle_connect 0x7fafd8036690 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249866645 0 0) 0x7fafd8030150 con 0x7fafd8038f10 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafcc004eb0 con 0x7fafd8038f10 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460129780 0 0) 0x7fafd800b2f0 con 0x7fafd80c91d0 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:43:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.237+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd8030150 con 0x7fafd80c91d0 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962095637 0 0) 0x7fafd80c1b20 con 0x7fafd8036690 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd800b2f0 con 0x7fafd8036690 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1185738868 0 0) 0x7fafcc004eb0 con 0x7fafd8038f10 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafd80c1b20 con 0x7fafd8038f10 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fafd4001900 con 0x7fafd8038f10 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient(hunting): dump: 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:24.245 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1206678136 0 0) 0x7fafd8030150 con 0x7fafd80c91d0 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafcc004eb0 con 0x7fafd80c91d0 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3211337044 0 0) 0x7fafd80c1b20 con 0x7fafd8038f10 2024-03-29T11:43:24.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 >> v1:172.21.15.155:6790/0 conn(0x7fafd8036690 legacy=0x7fafd80c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 >> v1:172.21.15.155:6789/0 conn(0x7fafd80c91d0 legacy=0x7fafd80097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 monclient: found mon.b 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd80c3fe0 con 0x7fafd8038f10 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafdd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafe5cf2640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafe5cf2640 10 monclient: _renew_subs 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafe5cf2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.238+0000 7fafe5cf2640 1 -- 172.21.15.155:0/3292895183 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd8009ec0 con 0x7fafd8038f10 2024-03-29T11:43:24.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafe5cf2640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafe5cf2640 10 monclient: _renew_subs 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafe5cf2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafe5cf2640 1 -- 172.21.15.155:0/3292895183 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd800a210 con 0x7fafd8038f10 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafe5cf2640 1 librados: init done 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafd4001a60 con 0x7fafd8038f10 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafdd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fafd4003810 con 0x7fafd8038f10 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafdd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafdd7fa640 10 monclient: dump: 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:24.248 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:24.249 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:24.249 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:24.249 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:24.249 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:24.249 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:24.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.239+0000 7fafdeffd640 4 set_mon_vals no callback set 2024-03-29T11:43:24.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.240+0000 7fafe5cf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:24.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.240+0000 7fafe5cf2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:24.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.240+0000 7fafe5cf2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:24.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.240+0000 7fafe5cf2640 1 -- 172.21.15.155:0/3292895183 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd8007b90 con 0x7fafd8038f10 2024-03-29T11:43:24.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.241+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fafd4003a90 con 0x7fafd8038f10 2024-03-29T11:43:24.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.242+0000 7fafe4cf0640 10 monclient: get_auth_request con 0x7fafcc06de80 auth_method 0 2024-03-29T11:43:24.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.243+0000 7fafdd7fa640 10 client.4489.objecter ms_handle_connect 0x7fafcc06de80 2024-03-29T11:43:24.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.243+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fafd4084990 con 0x7fafd8038f10 2024-03-29T11:43:24.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.243+0000 7fafdd7fa640 10 client.4489.objecter ms_dispatch 0x7fafe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:24.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.243+0000 7fafdd7fa640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:24.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.243+0000 7fafdd7fa640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:24.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.243+0000 7fafdd7fa640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-29T11:43:24.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.248+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafd4050270 con 0x7fafd8038f10 2024-03-29T11:43:24.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.248+0000 7fafdd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:24.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.248+0000 7fafdd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:24.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.609+0000 7fafe5cf2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:24.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.609+0000 7fafe5cf2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:24.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.609+0000 7fafe5cf2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:24.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.609+0000 7fafe5cf2640 1 -- 172.21.15.155:0/3292895183 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafd800b0f0 con 0x7fafd8038f10 2024-03-29T11:43:24.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.610+0000 7fafdd7fa640 1 -- 172.21.15.155:0/3292895183 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fafd40521a0 con 0x7fafd8038f10 2024-03-29T11:43:24.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.610+0000 7fafdd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:24.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.610+0000 7fafdd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:24.613 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:43:24.613 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:24.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.613+0000 7fafe5cf2640 10 librados: watch_flush enter 2024-03-29T11:43:24.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.613+0000 7fafe5cf2640 10 librados: watch_flush exit 2024-03-29T11:43:24.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.613+0000 7fafe5cf2640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-29T11:43:24.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.613+0000 7fafe5cf2640 10 client.4489.objecter successfully canceled tick 2024-03-29T11:43:24.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.614+0000 7fafe5cf2640 1 -- 172.21.15.155:0/3292895183 >> v1:172.21.15.155:6816/1760814708 conn(0x7fafcc06de80 legacy=0x7fafcc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:24.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.614+0000 7fafe5cf2640 10 monclient: shutdown 2024-03-29T11:43:24.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.614+0000 7fafe5cf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:24.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.614+0000 7fafe5cf2640 1 -- 172.21.15.155:0/3292895183 >> v1:172.21.15.183:6789/0 conn(0x7fafd8038f10 legacy=0x7fafd80c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:24.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.614+0000 7fafe5cf2640 1 -- 172.21.15.155:0/3292895183 shutdown_connections 2024-03-29T11:43:24.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.614+0000 7fafe5cf2640 1 -- 172.21.15.155:0/3292895183 >> 172.21.15.155:0/3292895183 conn(0x7fafd8005c20 msgr2=0x7fafd803b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:24.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.614+0000 7fafe5cf2640 1 -- 172.21.15.155:0/3292895183 shutdown_connections 2024-03-29T11:43:24.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.615+0000 7fafe5cf2640 1 -- 172.21.15.155:0/3292895183 wait complete. 2024-03-29T11:43:24.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:24.615+0000 7fafe5cf2640 1 librados: shutdown 2024-03-29T11:43:24.633 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:31.440 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:31.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.590+0000 7f2317ce2640 10 monclient: get_monmap_and_config 2024-03-29T11:43:31.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.590+0000 7f2317ce2640 10 monclient: build_initial_monmap 2024-03-29T11:43:31.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.590+0000 7f2317ce2640 10 monclient: monmap: 2024-03-29T11:43:31.593 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:31.593 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:31.593 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:43:31.590704+0000 2024-03-29T11:43:31.593 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:43:31.590704+0000 2024-03-29T11:43:31.594 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:31.594 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:31.594 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:31.594 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:43:31.594 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:43:31.594 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:31.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.591+0000 7f2317ce2640 1 Processor -- start 2024-03-29T11:43:31.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.591+0000 7f2317ce2640 1 -- start start 2024-03-29T11:43:31.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.591+0000 7f2317ce2640 10 monclient: init 2024-03-29T11:43:31.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.592+0000 7f2317ce2640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:31.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.592+0000 7f2317ce2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:31.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.592+0000 7f2317ce2640 10 monclient(hunting): picked mon.noname-b con 0x7f23100b2520 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:31.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.592+0000 7f2317ce2640 10 monclient(hunting): picked mon.noname-c con 0x7f23100b2900 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:31.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.592+0000 7f2317ce2640 10 monclient(hunting): picked mon.noname-a con 0x7f23100a9c70 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:31.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.592+0000 7f2317ce2640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2310142170 con 0x7f23100a9c70 2024-03-29T11:43:31.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.592+0000 7f2317ce2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23100b2260 con 0x7f23100b2900 2024-03-29T11:43:31.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.592+0000 7f2317ce2640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23100aaf60 con 0x7f23100b2520 2024-03-29T11:43:31.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.592+0000 7f2317ce2640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:31.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.592+0000 7f2317ce2640 10 monclient(hunting): authenticate will time out at 1393.718228s 2024-03-29T11:43:31.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.595+0000 7f2315a7e640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f23100b2520 0x7f23100b1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:42187/0 (socket says 172.21.15.155:42187) 2024-03-29T11:43:31.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.595+0000 7f2315a7e640 1 -- 172.21.15.155:0/1947400201 learned_addr learned my addr 172.21.15.155:0/1947400201 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:31.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.596+0000 7f2314a7c640 1 -- 172.21.15.155:0/1947400201 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874529850 0 0) 0x7f23100aaf60 con 0x7f23100b2520 2024-03-29T11:43:31.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.596+0000 7f2314a7c640 10 monclient(hunting): _negotiate 2024-03-29T11:43:31.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.596+0000 7f2314a7c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:31.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.596+0000 7f2314a7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:31.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.596+0000 7f2314a7c640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:43:31.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.596+0000 7f2314a7c640 1 -- 172.21.15.155:0/1947400201 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22fc003520 con 0x7f23100b2520 2024-03-29T11:43:31.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.598+0000 7f2314a7c640 1 -- 172.21.15.155:0/1947400201 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2675975954 0 0) 0x7f22fc003520 con 0x7f23100b2520 2024-03-29T11:43:31.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.598+0000 7f2314a7c640 1 -- 172.21.15.155:0/1947400201 >> v1:172.21.15.183:6789/0 conn(0x7f23100b2900 legacy=0x7f2310142600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:31.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 1 -- 172.21.15.155:0/1947400201 >> v1:172.21.15.155:6789/0 conn(0x7f23100a9c70 legacy=0x7f231014d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:31.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 10 monclient: _finish_hunting 0 2024-03-29T11:43:31.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 1 monclient: found mon.noname-b 2024-03-29T11:43:31.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:31.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:43:31.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 1 -- 172.21.15.155:0/1947400201 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2310142d10 con 0x7f23100b2520 2024-03-29T11:43:31.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 10 monclient: _finish_auth 0 2024-03-29T11:43:31.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 10 monclient: _check_auth_tickets 2024-03-29T11:43:31.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:31.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 1 -- 172.21.15.155:0/1947400201 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f230c001690 con 0x7f23100b2520 2024-03-29T11:43:31.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2314a7c640 10 monclient: dump: 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:31.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2317ce2640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:43:31.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.599+0000 7f2317ce2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:31.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.601+0000 7f2314a7c640 1 -- 172.21.15.155:0/1947400201 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f230c001920 con 0x7f23100b2520 2024-03-29T11:43:31.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.601+0000 7f2314a7c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:31.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.601+0000 7f2317ce2640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:31.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.601+0000 7f2317ce2640 4 set_mon_vals no callback set 2024-03-29T11:43:31.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.602+0000 7f2317ce2640 10 monclient: shutdown 2024-03-29T11:43:31.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.602+0000 7f2317ce2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:31.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.602+0000 7f2317ce2640 1 -- 172.21.15.155:0/1947400201 >> v1:172.21.15.155:6790/0 conn(0x7f23100b2520 legacy=0x7f23100b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:31.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.605+0000 7f2317ce2640 1 -- 172.21.15.155:0/1947400201 shutdown_connections 2024-03-29T11:43:31.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.605+0000 7f2317ce2640 1 -- 172.21.15.155:0/1947400201 >> 172.21.15.155:0/1947400201 conn(0x7f231006d180 msgr2=0x7f2310066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:31.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.605+0000 7f2317ce2640 1 -- 172.21.15.155:0/1947400201 shutdown_connections 2024-03-29T11:43:31.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.605+0000 7f2317ce2640 1 -- 172.21.15.155:0/1947400201 wait complete. 2024-03-29T11:43:31.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.606+0000 7f2317ce2640 10 monclient: build_initial_monmap 2024-03-29T11:43:31.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.606+0000 7f2317ce2640 10 monclient: monmap: 2024-03-29T11:43:31.610 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:31.610 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:31.610 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.606+0000 7f2317ce2640 1 librados: starting msgr at 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.606+0000 7f2317ce2640 1 librados: starting objecter 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.606+0000 7f2317ce2640 1 Processor -- start 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.606+0000 7f2317ce2640 1 -- start start 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.606+0000 7f2317ce2640 1 librados: setting wanted keys 2024-03-29T11:43:31.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.606+0000 7f2317ce2640 1 librados: calling monclient init 2024-03-29T11:43:31.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.606+0000 7f2317ce2640 10 monclient: init 2024-03-29T11:43:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.607+0000 7f2317ce2640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.607+0000 7f2317ce2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.607+0000 7f2317ce2640 10 monclient(hunting): picked mon.noname-b con 0x7f23100a9c70 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.607+0000 7f2317ce2640 10 monclient(hunting): picked mon.noname-c con 0x7f23100b2520 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.607+0000 7f2317ce2640 10 monclient(hunting): picked mon.noname-a con 0x7f231013bc20 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.607+0000 7f2317ce2640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231013df90 con 0x7f231013bc20 2024-03-29T11:43:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.607+0000 7f2317ce2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231013eb80 con 0x7f23100a9c70 2024-03-29T11:43:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.607+0000 7f2317ce2640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231013c150 con 0x7f23100b2520 2024-03-29T11:43:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.607+0000 7f2317ce2640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:31.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.607+0000 7f2317ce2640 10 monclient(hunting): authenticate will time out at 1393.733596s 2024-03-29T11:43:31.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.614+0000 7f231627f640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f231013bc20 0x7f23101e24e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:38247/0 (socket says 172.21.15.155:38247) 2024-03-29T11:43:31.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.614+0000 7f2315a7e640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f23100a9c70 0x7f231013e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:50577/0 (socket says 172.21.15.155:50577) 2024-03-29T11:43:31.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.614+0000 7f231627f640 1 -- 172.21.15.155:0/2163845167 learned_addr learned my addr 172.21.15.155:0/2163845167 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:31.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.614+0000 7f23067fc640 10 client.?.objecter ms_handle_connect 0x7f231013bc20 2024-03-29T11:43:31.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.614+0000 7f23067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:31.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.615+0000 7f23067fc640 10 client.?.objecter ms_handle_connect 0x7f23100a9c70 2024-03-29T11:43:31.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.615+0000 7f23067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:31.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.615+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668693335 0 0) 0x7f231013df90 con 0x7f231013bc20 2024-03-29T11:43:31.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.615+0000 7f23067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:31.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.615+0000 7f23067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:31.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.615+0000 7f23067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:31.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.615+0000 7f23067fc640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:43:31.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.615+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f4004eb0 con 0x7f231013bc20 2024-03-29T11:43:31.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.615+0000 7f23067fc640 10 client.?.objecter ms_handle_connect 0x7f23100b2520 2024-03-29T11:43:31.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.615+0000 7f23067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:31.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997680694 0 0) 0x7f231013c150 con 0x7f23100b2520 2024-03-29T11:43:31.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:43:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f40051d0 con 0x7f23100b2520 2024-03-29T11:43:31.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3143506027 0 0) 0x7f22f40051d0 con 0x7f23100b2520 2024-03-29T11:43:31.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22f4005640 con 0x7f23100b2520 2024-03-29T11:43:31.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2308001ac0 con 0x7f23100b2520 2024-03-29T11:43:31.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:31.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:31.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 10 monclient(hunting): dump: 2024-03-29T11:43:31.624 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:31.624 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:31.624 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:31.624 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:31.624 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:31.625 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:31.625 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:31.625 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:31.625 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:31.625 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:31.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.616+0000 7f23067fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:31.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.617+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560521776 0 0) 0x7f231013eb80 con 0x7f23100a9c70 2024-03-29T11:43:31.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.617+0000 7f23067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:31.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.617+0000 7f23067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:31.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.617+0000 7f23067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:31.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.617+0000 7f23067fc640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:43:31.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.617+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f4005b90 con 0x7f23100a9c70 2024-03-29T11:43:31.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 295376425 0 0) 0x7f22f4005640 con 0x7f23100b2520 2024-03-29T11:43:31.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 >> v1:172.21.15.183:6789/0 conn(0x7f23100a9c70 legacy=0x7f231013e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:31.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 >> v1:172.21.15.155:6789/0 conn(0x7f231013bc20 legacy=0x7f23101e24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:31.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f23067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:31.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f23067fc640 1 monclient: found mon.c 2024-03-29T11:43:31.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f23067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:31.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f23067fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:31.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231013db10 con 0x7f23100b2520 2024-03-29T11:43:31.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f23067fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:31.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f23067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:31.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f23067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:31.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f2317ce2640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:43:31.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f2317ce2640 10 monclient: _renew_subs 2024-03-29T11:43:31.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f2317ce2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:31.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f2317ce2640 1 -- 172.21.15.155:0/2163845167 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2310144e00 con 0x7f23100b2520 2024-03-29T11:43:31.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f2317ce2640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:31.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f2317ce2640 10 monclient: _renew_subs 2024-03-29T11:43:31.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f2317ce2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:31.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f2317ce2640 1 -- 172.21.15.155:0/2163845167 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23101372d0 con 0x7f23100b2520 2024-03-29T11:43:31.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.618+0000 7f2317ce2640 1 librados: init done 2024-03-29T11:43:31.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.620+0000 7f22fbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:31.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.620+0000 7f22fbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:31.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.620+0000 7f22fbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:31.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.620+0000 7f22fbfff640 1 -- 172.21.15.155:0/2163845167 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f0005080 con 0x7f23100b2520 2024-03-29T11:43:31.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.622+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2308001fd0 con 0x7f23100b2520 2024-03-29T11:43:31.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.622+0000 7f23067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:31.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.622+0000 7f2307fff640 4 set_mon_vals no callback set 2024-03-29T11:43:31.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.623+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f2308001ac0 con 0x7f23100b2520 2024-03-29T11:43:31.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.623+0000 7f23067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:31.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.623+0000 7f23067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:31.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.623+0000 7f23067fc640 10 monclient: dump: 2024-03-29T11:43:31.632 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:31.632 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:31.632 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:31.632 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:31.632 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:31.633 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:31.633 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:31.633 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:31.633 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:31.633 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:31.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.624+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f2308019810 con 0x7f23100b2520 2024-03-29T11:43:31.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.624+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f2308083680 con 0x7f23100b2520 2024-03-29T11:43:31.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.624+0000 7f23067fc640 10 client.4598.objecter ms_dispatch 0x7f2310000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:31.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.624+0000 7f23067fc640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:31.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.624+0000 7f23067fc640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:31.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.624+0000 7f23067fc640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:43:31.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.625+0000 7f2315a7e640 10 monclient: get_auth_request con 0x7f22f406e620 auth_method 0 2024-03-29T11:43:31.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.625+0000 7f23067fc640 10 client.4598.objecter ms_handle_connect 0x7f22f406e620 2024-03-29T11:43:31.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.630+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f230804f020 con 0x7f23100b2520 2024-03-29T11:43:31.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.630+0000 7f23067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:31.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.630+0000 7f23067fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:31.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.966+0000 7f22fbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:31.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.966+0000 7f22fbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:31.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.966+0000 7f22fbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:31.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.966+0000 7f22fbfff640 1 -- 172.21.15.155:0/2163845167 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22f0005330 con 0x7f23100b2520 2024-03-29T11:43:31.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.970+0000 7f23067fc640 1 -- 172.21.15.155:0/2163845167 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f230804f020 con 0x7f23100b2520 2024-03-29T11:43:31.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.970+0000 7f23067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:31.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.970+0000 7f23067fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:31.973 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:43:31.974 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:31.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.973+0000 7f22fbfff640 10 librados: watch_flush enter 2024-03-29T11:43:31.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.975+0000 7f22fbfff640 10 librados: watch_flush exit 2024-03-29T11:43:31.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.975+0000 7f22fbfff640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:43:31.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.975+0000 7f22fbfff640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:43:31.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.975+0000 7f22fbfff640 1 -- 172.21.15.155:0/2163845167 >> v1:172.21.15.155:6816/1760814708 conn(0x7f22f406e620 legacy=0x7f22f4070ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:31.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.975+0000 7f22fbfff640 10 monclient: shutdown 2024-03-29T11:43:31.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.975+0000 7f22fbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:31.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.975+0000 7f22fbfff640 1 -- 172.21.15.155:0/2163845167 >> v1:172.21.15.155:6790/0 conn(0x7f23100b2520 legacy=0x7f2310136bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:31.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.975+0000 7f22fbfff640 1 -- 172.21.15.155:0/2163845167 shutdown_connections 2024-03-29T11:43:31.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.975+0000 7f22fbfff640 1 -- 172.21.15.155:0/2163845167 >> 172.21.15.155:0/2163845167 conn(0x7f231006d180 msgr2=0x7f2310066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:31.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.975+0000 7f22fbfff640 1 -- 172.21.15.155:0/2163845167 shutdown_connections 2024-03-29T11:43:31.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.976+0000 7f22fbfff640 1 -- 172.21.15.155:0/2163845167 wait complete. 2024-03-29T11:43:31.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:31.976+0000 7f22fbfff640 1 librados: shutdown 2024-03-29T11:43:31.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:38.806 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:38.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.954+0000 7f514e463640 10 monclient: get_monmap_and_config 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.954+0000 7f514e463640 10 monclient: build_initial_monmap 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.954+0000 7f514e463640 10 monclient: monmap: 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:43:38.956350+0000 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:43:38.956350+0000 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:43:38.960 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:38.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.957+0000 7f514e463640 1 Processor -- start 2024-03-29T11:43:38.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.957+0000 7f514e463640 1 -- start start 2024-03-29T11:43:38.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.957+0000 7f514e463640 10 monclient: init 2024-03-29T11:43:38.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.958+0000 7f514e463640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:38.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.958+0000 7f514e463640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:38.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.958+0000 7f514e463640 10 monclient(hunting): picked mon.noname-a con 0x7f51480ae250 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:38.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.958+0000 7f514e463640 10 monclient(hunting): picked mon.noname-b con 0x7f514813f970 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:38.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.958+0000 7f514e463640 10 monclient(hunting): picked mon.noname-c con 0x7f5148142800 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:38.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.958+0000 7f514e463640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51480afe00 con 0x7f51480ae250 2024-03-29T11:43:38.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.958+0000 7f514e463640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5148142c10 con 0x7f5148142800 2024-03-29T11:43:38.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.958+0000 7f514e463640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51480afb20 con 0x7f514813f970 2024-03-29T11:43:38.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.958+0000 7f514e463640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:38.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.958+0000 7f514e463640 10 monclient(hunting): authenticate will time out at 1401.085626s 2024-03-29T11:43:38.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5147fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f51480ae250 0x7f514813f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:59363/0 (socket says 172.21.15.155:59363) 2024-03-29T11:43:38.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5147fff640 1 -- 172.21.15.155:0/515157028 learned_addr learned my addr 172.21.15.155:0/515157028 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 1 -- 172.21.15.155:0/515157028 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172095552 0 0) 0x7f51480afe00 con 0x7f51480ae250 2024-03-29T11:43:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:43:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 1 -- 172.21.15.155:0/515157028 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5130003520 con 0x7f51480ae250 2024-03-29T11:43:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 1 -- 172.21.15.155:0/515157028 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453631148 0 0) 0x7f51480afb20 con 0x7f514813f970 2024-03-29T11:43:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.961+0000 7f5146ffd640 1 -- 172.21.15.155:0/515157028 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5130003840 con 0x7f514813f970 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 1 -- 172.21.15.155:0/515157028 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1085134065 0 0) 0x7f5130003840 con 0x7f514813f970 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 1 -- 172.21.15.155:0/515157028 >> v1:172.21.15.183:6789/0 conn(0x7f5148142800 legacy=0x7f514814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 1 -- 172.21.15.155:0/515157028 >> v1:172.21.15.155:6789/0 conn(0x7f51480ae250 legacy=0x7f514813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 1 monclient: found mon.noname-b 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 1 -- 172.21.15.155:0/515157028 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51480aab80 con 0x7f514813f970 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 1 -- 172.21.15.155:0/515157028 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f513c001670 con 0x7f514813f970 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:38.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f5146ffd640 10 monclient: dump: 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f514e463640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:43:38.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.962+0000 7f514e463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:38.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.964+0000 7f5146ffd640 1 -- 172.21.15.155:0/515157028 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f513c001850 con 0x7f514813f970 2024-03-29T11:43:38.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.964+0000 7f5146ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:38.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.964+0000 7f5146ffd640 1 -- 172.21.15.155:0/515157028 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f513c001670 con 0x7f514813f970 2024-03-29T11:43:38.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.964+0000 7f5146ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:38.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.964+0000 7f5146ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:38.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.964+0000 7f5146ffd640 10 monclient: dump: 2024-03-29T11:43:38.972 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:38.972 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:38.972 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:38.972 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:38.972 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.966+0000 7f514e463640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.966+0000 7f514e463640 4 set_mon_vals no callback set 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.967+0000 7f514e463640 10 monclient: shutdown 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.967+0000 7f514e463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.967+0000 7f514e463640 1 -- 172.21.15.155:0/515157028 >> v1:172.21.15.155:6790/0 conn(0x7f514813f970 legacy=0x7f51481420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.967+0000 7f514e463640 1 -- 172.21.15.155:0/515157028 shutdown_connections 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.967+0000 7f514e463640 1 -- 172.21.15.155:0/515157028 >> 172.21.15.155:0/515157028 conn(0x7f51480b35e0 msgr2=0x7f51480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.967+0000 7f514e463640 1 -- 172.21.15.155:0/515157028 shutdown_connections 2024-03-29T11:43:38.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.967+0000 7f514e463640 1 -- 172.21.15.155:0/515157028 wait complete. 2024-03-29T11:43:38.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.968+0000 7f514e463640 10 monclient: build_initial_monmap 2024-03-29T11:43:38.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.968+0000 7f514e463640 10 monclient: monmap: 2024-03-29T11:43:38.974 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:38.974 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:38.974 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.969+0000 7f514e463640 1 librados: starting msgr at 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.969+0000 7f514e463640 1 librados: starting objecter 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.969+0000 7f514e463640 1 Processor -- start 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.970+0000 7f514e463640 1 -- start start 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.970+0000 7f514e463640 1 librados: setting wanted keys 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.970+0000 7f514e463640 1 librados: calling monclient init 2024-03-29T11:43:38.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.970+0000 7f514e463640 10 monclient: init 2024-03-29T11:43:38.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f514e463640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:38.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f514e463640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:38.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f514e463640 10 monclient(hunting): picked mon.noname-a con 0x7f51480ae250 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:38.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f514e463640 10 monclient(hunting): picked mon.noname-c con 0x7f514813f970 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:38.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f514e463640 10 monclient(hunting): picked mon.noname-b con 0x7f5148142800 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:38.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f514e463640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51480711e0 con 0x7f51480ae250 2024-03-29T11:43:38.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f514e463640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5148071410 con 0x7f5148142800 2024-03-29T11:43:38.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f514e463640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51480716f0 con 0x7f514813f970 2024-03-29T11:43:38.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f514e463640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:38.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f514e463640 10 monclient(hunting): authenticate will time out at 1401.098181s 2024-03-29T11:43:38.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f51477fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f514813f970 0x7f514806f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:47156/0 (socket says 172.21.15.155:47156) 2024-03-29T11:43:38.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f51477fe640 1 -- 172.21.15.155:0/3316791177 learned_addr learned my addr 172.21.15.155:0/3316791177 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:38.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f5144ff9640 10 client.?.objecter ms_handle_connect 0x7f514813f970 2024-03-29T11:43:38.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.971+0000 7f5144ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:38.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.973+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026959483 0 0) 0x7f51480716f0 con 0x7f514813f970 2024-03-29T11:43:38.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.973+0000 7f5144ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:38.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.973+0000 7f5144ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:38.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.973+0000 7f5144ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:38.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.973+0000 7f5144ff9640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:43:38.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.973+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5134004eb0 con 0x7f514813f970 2024-03-29T11:43:38.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 client.?.objecter ms_handle_connect 0x7f5148142800 2024-03-29T11:43:38.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:38.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 client.?.objecter ms_handle_connect 0x7f51480ae250 2024-03-29T11:43:38.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:38.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462846482 0 0) 0x7f5148071410 con 0x7f5148142800 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51480716f0 con 0x7f5148142800 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982924853 0 0) 0x7f51480711e0 con 0x7f51480ae250 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.975+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5148071410 con 0x7f51480ae250 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4200407493 0 0) 0x7f5134004eb0 con 0x7f514813f970 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51480711e0 con 0x7f514813f970 2024-03-29T11:43:38.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f513c001670 con 0x7f514813f970 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:38.987 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 60445372 0 0) 0x7f5148071410 con 0x7f51480ae250 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5134004eb0 con 0x7f51480ae250 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f51380012b0 con 0x7f51480ae250 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:38.988 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1411074327 0 0) 0x7f5134004eb0 con 0x7f51480ae250 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 >> v1:172.21.15.155:6790/0 conn(0x7f514813f970 legacy=0x7f514806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 >> v1:172.21.15.183:6789/0 conn(0x7f5148142800 legacy=0x7f514806fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 monclient: found mon.a 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51480719d0 con 0x7f51480ae250 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:38.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f514e463640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f514e463640 10 monclient: _renew_subs 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f514e463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f514e463640 1 -- 172.21.15.155:0/3316791177 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5148070810 con 0x7f51480ae250 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f514e463640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f514e463640 10 monclient: _renew_subs 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f514e463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f514e463640 1 -- 172.21.15.155:0/3316791177 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5148070370 con 0x7f51480ae250 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f514e463640 1 librados: init done 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5138001d10 con 0x7f51480ae250 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f5144ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.976+0000 7f51467fc640 4 set_mon_vals no callback set 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.978+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f5138004980 con 0x7f51480ae250 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.978+0000 7f5144ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.978+0000 7f5144ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.979+0000 7f5144ff9640 10 monclient: dump: 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.979+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f5138004b20 con 0x7f51480ae250 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.979+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f51380848f0 con 0x7f51480ae250 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.979+0000 7f51477fe640 10 monclient: get_auth_request con 0x7f513406df40 auth_method 0 2024-03-29T11:43:38.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.979+0000 7f5144ff9640 10 client.4545.objecter ms_dispatch 0x7f5148000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:38.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.979+0000 7f5144ff9640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:38.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.979+0000 7f5144ff9640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:38.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.980+0000 7f5144ff9640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-29T11:43:38.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.980+0000 7f5144ff9640 10 client.4545.objecter ms_handle_connect 0x7f513406df40 2024-03-29T11:43:38.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.981+0000 7f514e463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:38.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.981+0000 7f514e463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:38.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.981+0000 7f514e463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:43:38.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.981+0000 7f514e463640 1 -- 172.21.15.155:0/3316791177 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5128005080 con 0x7f51480ae250 2024-03-29T11:43:38.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.990+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5138050080 con 0x7f51480ae250 2024-03-29T11:43:38.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.990+0000 7f5144ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:38.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:38.990+0000 7f5144ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:39.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.310+0000 7f514e463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:39.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.310+0000 7f514e463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:39.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.310+0000 7f514e463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:43:39.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.310+0000 7f514e463640 1 -- 172.21.15.155:0/3316791177 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5128005330 con 0x7f51480ae250 2024-03-29T11:43:39.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.314+0000 7f5144ff9640 1 -- 172.21.15.155:0/3316791177 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f5138052180 con 0x7f51480ae250 2024-03-29T11:43:39.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.314+0000 7f5144ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:39.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.314+0000 7f5144ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:39.319 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:43:39.319 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.318+0000 7f514e463640 10 librados: watch_flush enter 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.318+0000 7f514e463640 10 librados: watch_flush exit 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.318+0000 7f514e463640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.318+0000 7f514e463640 10 client.4545.objecter successfully canceled tick 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.318+0000 7f514e463640 1 -- 172.21.15.155:0/3316791177 >> v1:172.21.15.155:6816/1760814708 conn(0x7f513406df40 legacy=0x7f5134070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.318+0000 7f514e463640 10 monclient: shutdown 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.318+0000 7f514e463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.318+0000 7f514e463640 1 -- 172.21.15.155:0/3316791177 >> v1:172.21.15.155:6789/0 conn(0x7f51480ae250 legacy=0x7f514813ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.319+0000 7f514e463640 1 -- 172.21.15.155:0/3316791177 shutdown_connections 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.319+0000 7f514e463640 1 -- 172.21.15.155:0/3316791177 >> 172.21.15.155:0/3316791177 conn(0x7f51480b35e0 msgr2=0x7f5148142000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.319+0000 7f514e463640 1 -- 172.21.15.155:0/3316791177 shutdown_connections 2024-03-29T11:43:39.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.319+0000 7f514e463640 1 -- 172.21.15.155:0/3316791177 wait complete. 2024-03-29T11:43:39.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:39.319+0000 7f514e463640 1 librados: shutdown 2024-03-29T11:43:39.342 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:46.150 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:46.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.297+0000 7f85429bc640 10 monclient: get_monmap_and_config 2024-03-29T11:43:46.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.297+0000 7f85429bc640 10 monclient: build_initial_monmap 2024-03-29T11:43:46.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.297+0000 7f85429bc640 10 monclient: monmap: 2024-03-29T11:43:46.302 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:46.302 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:46.302 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:43:46.298873+0000 2024-03-29T11:43:46.302 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:43:46.298873+0000 2024-03-29T11:43:46.302 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:46.302 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:46.302 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:46.302 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:43:46.302 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:43:46.302 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:46.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.298+0000 7f85429bc640 1 Processor -- start 2024-03-29T11:43:46.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.298+0000 7f85429bc640 1 -- start start 2024-03-29T11:43:46.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.298+0000 7f85429bc640 10 monclient: init 2024-03-29T11:43:46.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.299+0000 7f85429bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:46.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.299+0000 7f85429bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:46.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.299+0000 7f85429bc640 10 monclient(hunting): picked mon.noname-c con 0x7f8534006d50 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:46.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.300+0000 7f85429bc640 10 monclient(hunting): picked mon.noname-b con 0x7f85340c4dc0 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:46.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.300+0000 7f85429bc640 10 monclient(hunting): picked mon.noname-a con 0x7f85340cb5b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:46.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.300+0000 7f85429bc640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8534008d50 con 0x7f85340cb5b0 2024-03-29T11:43:46.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.300+0000 7f85429bc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8534007130 con 0x7f8534006d50 2024-03-29T11:43:46.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.300+0000 7f85429bc640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8534030280 con 0x7f85340c4dc0 2024-03-29T11:43:46.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.300+0000 7f85429bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:46.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.300+0000 7f85429bc640 10 monclient(hunting): authenticate will time out at 1408.427117s 2024-03-29T11:43:46.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.300+0000 7f85411b9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f85340c4dc0 0x7f85340caea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33340/0 (socket says 172.21.15.155:33340) 2024-03-29T11:43:46.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.300+0000 7f85411b9640 1 -- 172.21.15.155:0/525867451 learned_addr learned my addr 172.21.15.155:0/525867451 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.304+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917431496 0 0) 0x7f8534030280 con 0x7f85340c4dc0 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.304+0000 7f85409b8640 10 monclient(hunting): _negotiate 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.304+0000 7f85409b8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.304+0000 7f85409b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.304+0000 7f85409b8640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.304+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8528003520 con 0x7f85340c4dc0 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213915399 0 0) 0x7f8534007130 con 0x7f8534006d50 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient(hunting): _negotiate 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:43:46.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8534030280 con 0x7f8534006d50 2024-03-29T11:43:46.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077703742 0 0) 0x7f8534008d50 con 0x7f85340cb5b0 2024-03-29T11:43:46.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient(hunting): _negotiate 2024-03-29T11:43:46.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:46.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:46.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:43:46.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8534007130 con 0x7f85340cb5b0 2024-03-29T11:43:46.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 155457571 0 0) 0x7f8534030280 con 0x7f8534006d50 2024-03-29T11:43:46.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 >> v1:172.21.15.155:6790/0 conn(0x7f85340c4dc0 legacy=0x7f85340caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:46.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 >> v1:172.21.15.155:6789/0 conn(0x7f85340cb5b0 legacy=0x7f85340d3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:46.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient: _finish_hunting 0 2024-03-29T11:43:46.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 1 monclient: found mon.noname-c 2024-03-29T11:43:46.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:46.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:43:46.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85340304f0 con 0x7f8534006d50 2024-03-29T11:43:46.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient: _finish_auth 0 2024-03-29T11:43:46.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient: _check_auth_tickets 2024-03-29T11:43:46.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:46.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8538001670 con 0x7f8534006d50 2024-03-29T11:43:46.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:46.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:46.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85409b8640 10 monclient: dump: 2024-03-29T11:43:46.314 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:46.314 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:46.314 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:46.314 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:46.314 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:46.315 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:46.315 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:46.315 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:46.315 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:46.315 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:46.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85429bc640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:43:46.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.306+0000 7f85429bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:46.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.307+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85380017d0 con 0x7f8534006d50 2024-03-29T11:43:46.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.307+0000 7f85409b8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:46.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.307+0000 7f85409b8640 1 -- 172.21.15.155:0/525867451 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8538001ba0 con 0x7f8534006d50 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.307+0000 7f85409b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.307+0000 7f85409b8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.307+0000 7f85409b8640 10 monclient: dump: 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.307+0000 7f85429bc640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:46.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.307+0000 7f85429bc640 4 set_mon_vals no callback set 2024-03-29T11:43:46.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.308+0000 7f85429bc640 10 monclient: shutdown 2024-03-29T11:43:46.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.308+0000 7f85429bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:46.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.308+0000 7f85429bc640 1 -- 172.21.15.155:0/525867451 >> v1:172.21.15.183:6789/0 conn(0x7f8534006d50 legacy=0x7f85340c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:46.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.310+0000 7f85429bc640 1 -- 172.21.15.155:0/525867451 shutdown_connections 2024-03-29T11:43:46.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.310+0000 7f85429bc640 1 -- 172.21.15.155:0/525867451 >> 172.21.15.155:0/525867451 conn(0x7f8534005bd0 msgr2=0x7f8534005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:46.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.311+0000 7f85429bc640 1 -- 172.21.15.155:0/525867451 shutdown_connections 2024-03-29T11:43:46.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.311+0000 7f85429bc640 1 -- 172.21.15.155:0/525867451 wait complete. 2024-03-29T11:43:46.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.312+0000 7f85429bc640 10 monclient: build_initial_monmap 2024-03-29T11:43:46.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.312+0000 7f85429bc640 10 monclient: monmap: 2024-03-29T11:43:46.319 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:46.319 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:46.319 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.312+0000 7f85429bc640 1 librados: starting msgr at 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.312+0000 7f85429bc640 1 librados: starting objecter 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.312+0000 7f85429bc640 1 Processor -- start 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.312+0000 7f85429bc640 1 -- start start 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.312+0000 7f85429bc640 1 librados: setting wanted keys 2024-03-29T11:43:46.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.312+0000 7f85429bc640 1 librados: calling monclient init 2024-03-29T11:43:46.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.312+0000 7f85429bc640 10 monclient: init 2024-03-29T11:43:46.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.313+0000 7f85429bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:46.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.313+0000 7f85429bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:46.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.315+0000 7f85429bc640 10 monclient(hunting): picked mon.noname-a con 0x7f8534006d50 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:46.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.315+0000 7f85429bc640 10 monclient(hunting): picked mon.noname-c con 0x7f85340c4dc0 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:46.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.316+0000 7f85429bc640 10 monclient(hunting): picked mon.noname-b con 0x7f85340cb5b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:46.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.316+0000 7f85429bc640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8534031ea0 con 0x7f8534006d50 2024-03-29T11:43:46.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.316+0000 7f85429bc640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8534033bf0 con 0x7f85340cb5b0 2024-03-29T11:43:46.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.316+0000 7f85429bc640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853415fbc0 con 0x7f85340c4dc0 2024-03-29T11:43:46.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.316+0000 7f85429bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:46.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.316+0000 7f85429bc640 10 monclient(hunting): authenticate will time out at 1408.442925s 2024-03-29T11:43:46.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.316+0000 7f85411b9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f85340c4dc0 0x7f8534031790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36302/0 (socket says 172.21.15.155:36302) 2024-03-29T11:43:46.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.316+0000 7f85411b9640 1 -- 172.21.15.155:0/2552676508 learned_addr learned my addr 172.21.15.155:0/2552676508 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:46.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.317+0000 7f85327fc640 10 client.?.objecter ms_handle_connect 0x7f85340c4dc0 2024-03-29T11:43:46.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.317+0000 7f85327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:46.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.318+0000 7f85327fc640 10 client.?.objecter ms_handle_connect 0x7f8534006d50 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.318+0000 7f85327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.319+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861516685 0 0) 0x7f8534031ea0 con 0x7f8534006d50 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.319+0000 7f85327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.319+0000 7f85327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.319+0000 7f85327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.319+0000 7f85327fc640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.319+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8520004eb0 con 0x7f8534006d50 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.319+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978439503 0 0) 0x7f853415fbc0 con 0x7f85340c4dc0 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:46.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8534031ea0 con 0x7f85340c4dc0 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2556676986 0 0) 0x7f8534031ea0 con 0x7f85340c4dc0 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f853415fbc0 con 0x7f85340c4dc0 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f852c001950 con 0x7f85340c4dc0 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 10 monclient(hunting): dump: 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:46.331 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 389747215 0 0) 0x7f853415fbc0 con 0x7f85340c4dc0 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.320+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 >> v1:172.21.15.183:6789/0 conn(0x7f85340cb5b0 legacy=0x7f85341600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 >> v1:172.21.15.155:6789/0 conn(0x7f8534006d50 legacy=0x7f8534032510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 1 monclient: found mon.c 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85341607f0 con 0x7f85340c4dc0 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:46.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 10 client.?.objecter ms_handle_connect 0x7f85340cb5b0 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f852c001d60 con 0x7f85340c4dc0 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f852c0049c0 con 0x7f85340c4dc0 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85327fc640 10 monclient: dump: 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:46.333 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f8533fff640 4 set_mon_vals no callback set 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85429bc640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85429bc640 10 monclient: _renew_subs 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85429bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.321+0000 7f85429bc640 1 -- 172.21.15.155:0/2552676508 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8534167ec0 con 0x7f85340c4dc0 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.322+0000 7f85429bc640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.322+0000 7f85429bc640 10 monclient: _renew_subs 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.322+0000 7f85429bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:46.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.322+0000 7f85429bc640 1 -- 172.21.15.155:0/2552676508 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8534168230 con 0x7f85340c4dc0 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.322+0000 7f85429bc640 1 librados: init done 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.323+0000 7f85429bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.324+0000 7f85429bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.324+0000 7f85429bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.324+0000 7f85429bc640 1 -- 172.21.15.155:0/2552676508 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85340050f0 con 0x7f85340c4dc0 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.324+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f852c004b20 con 0x7f85340c4dc0 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.325+0000 7f85419ba640 10 monclient: get_auth_request con 0x7f852006db10 auth_method 0 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.325+0000 7f85327fc640 10 client.4610.objecter ms_handle_connect 0x7f852006db10 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.325+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f852c0849f0 con 0x7f85340c4dc0 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.325+0000 7f85327fc640 10 client.4610.objecter ms_dispatch 0x7f853c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.325+0000 7f85327fc640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.325+0000 7f85327fc640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:46.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.325+0000 7f85327fc640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-29T11:43:46.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.332+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f852c050100 con 0x7f85340c4dc0 2024-03-29T11:43:46.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.332+0000 7f85327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:46.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.332+0000 7f85327fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:46.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.652+0000 7f85429bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:46.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.652+0000 7f85429bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.652+0000 7f85429bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:43:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.652+0000 7f85429bc640 1 -- 172.21.15.155:0/2552676508 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f853415fd50 con 0x7f85340c4dc0 2024-03-29T11:43:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.652+0000 7f85327fc640 1 -- 172.21.15.155:0/2552676508 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f852c052200 con 0x7f85340c4dc0 2024-03-29T11:43:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.652+0000 7f85327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.652+0000 7f85327fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:46.657 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:43:46.657 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.656+0000 7f85429bc640 10 librados: watch_flush enter 2024-03-29T11:43:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.656+0000 7f85429bc640 10 librados: watch_flush exit 2024-03-29T11:43:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.656+0000 7f85429bc640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-29T11:43:46.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.656+0000 7f85429bc640 10 client.4610.objecter successfully canceled tick 2024-03-29T11:43:46.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.656+0000 7f85429bc640 1 -- 172.21.15.155:0/2552676508 >> v1:172.21.15.155:6816/1760814708 conn(0x7f852006db10 legacy=0x7f852006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:46.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.656+0000 7f85429bc640 10 monclient: shutdown 2024-03-29T11:43:46.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.656+0000 7f85429bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:46.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.656+0000 7f85429bc640 1 -- 172.21.15.155:0/2552676508 >> v1:172.21.15.155:6790/0 conn(0x7f85340c4dc0 legacy=0x7f8534031790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:46.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.657+0000 7f85429bc640 1 -- 172.21.15.155:0/2552676508 shutdown_connections 2024-03-29T11:43:46.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.657+0000 7f85429bc640 1 -- 172.21.15.155:0/2552676508 >> 172.21.15.155:0/2552676508 conn(0x7f8534005bd0 msgr2=0x7f85340c2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:46.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.657+0000 7f85429bc640 1 -- 172.21.15.155:0/2552676508 shutdown_connections 2024-03-29T11:43:46.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.658+0000 7f85429bc640 1 -- 172.21.15.155:0/2552676508 wait complete. 2024-03-29T11:43:46.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:46.658+0000 7f85429bc640 1 librados: shutdown 2024-03-29T11:43:46.678 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:53.485 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:53.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.625+0000 7fd63a930640 10 monclient: get_monmap_and_config 2024-03-29T11:43:53.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.625+0000 7fd63a930640 10 monclient: build_initial_monmap 2024-03-29T11:43:53.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.625+0000 7fd63a930640 10 monclient: monmap: 2024-03-29T11:43:53.630 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:53.630 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:53.630 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:43:53.627212+0000 2024-03-29T11:43:53.630 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:43:53.627212+0000 2024-03-29T11:43:53.630 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:53.630 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:53.630 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:53.631 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:43:53.631 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:43:53.631 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:53.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.627+0000 7fd63a930640 1 Processor -- start 2024-03-29T11:43:53.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.627+0000 7fd63a930640 1 -- start start 2024-03-29T11:43:53.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.627+0000 7fd63a930640 10 monclient: init 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd63a930640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd63a930640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd63a930640 10 monclient(hunting): picked mon.noname-a con 0x7fd6341457f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd63a930640 10 monclient(hunting): picked mon.noname-b con 0x7fd63413f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd63a930640 10 monclient(hunting): picked mon.noname-c con 0x7fd6340a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd63a930640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6340aaef0 con 0x7fd6341457f0 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd63a930640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd634145bd0 con 0x7fd6340a9d70 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd63a930640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd634145eb0 con 0x7fd63413f650 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd63a930640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd63a930640 10 monclient(hunting): authenticate will time out at 1415.755616s 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd6337fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fd63413f650 0x7fd6340b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:57274/0 (socket says 172.21.15.155:57274) 2024-03-29T11:43:53.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.628+0000 7fd6337fe640 1 -- 172.21.15.155:0/3235152476 learned_addr learned my addr 172.21.15.155:0/3235152476 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:53.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 1 -- 172.21.15.155:0/3235152476 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243071408 0 0) 0x7fd634145eb0 con 0x7fd63413f650 2024-03-29T11:43:53.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 1 -- 172.21.15.155:0/3235152476 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd61c003520 con 0x7fd63413f650 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 1 -- 172.21.15.155:0/3235152476 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184959483 0 0) 0x7fd6340aaef0 con 0x7fd6341457f0 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.629+0000 7fd632ffd640 1 -- 172.21.15.155:0/3235152476 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd634145eb0 con 0x7fd6341457f0 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 1 -- 172.21.15.155:0/3235152476 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1476842074 0 0) 0x7fd61c003520 con 0x7fd63413f650 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 1 -- 172.21.15.155:0/3235152476 >> v1:172.21.15.183:6789/0 conn(0x7fd6340a9d70 legacy=0x7fd63414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 1 -- 172.21.15.155:0/3235152476 >> v1:172.21.15.155:6789/0 conn(0x7fd6341457f0 legacy=0x7fd63413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:53.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 1 monclient: found mon.noname-b 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 1 -- 172.21.15.155:0/3235152476 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd634146340 con 0x7fd63413f650 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 1 -- 172.21.15.155:0/3235152476 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd628001690 con 0x7fd63413f650 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 10 monclient: dump: 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.637 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd63a930640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd63a930640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 1 -- 172.21.15.155:0/3235152476 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6280017f0 con 0x7fd63413f650 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 1 -- 172.21.15.155:0/3235152476 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd628001c40 con 0x7fd63413f650 2024-03-29T11:43:53.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd632ffd640 10 monclient: dump: 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd63a930640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:53.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.630+0000 7fd63a930640 4 set_mon_vals no callback set 2024-03-29T11:43:53.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.631+0000 7fd63a930640 10 monclient: shutdown 2024-03-29T11:43:53.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.631+0000 7fd63a930640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:53.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.631+0000 7fd63a930640 1 -- 172.21.15.155:0/3235152476 >> v1:172.21.15.155:6790/0 conn(0x7fd63413f650 legacy=0x7fd6340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:53.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.631+0000 7fd63a930640 1 -- 172.21.15.155:0/3235152476 shutdown_connections 2024-03-29T11:43:53.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.631+0000 7fd63a930640 1 -- 172.21.15.155:0/3235152476 >> 172.21.15.155:0/3235152476 conn(0x7fd634066a00 msgr2=0x7fd634066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:53.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.631+0000 7fd63a930640 1 -- 172.21.15.155:0/3235152476 shutdown_connections 2024-03-29T11:43:53.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.631+0000 7fd63a930640 1 -- 172.21.15.155:0/3235152476 wait complete. 2024-03-29T11:43:53.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.632+0000 7fd63a930640 10 monclient: build_initial_monmap 2024-03-29T11:43:53.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.632+0000 7fd63a930640 10 monclient: monmap: 2024-03-29T11:43:53.647 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:43:53.647 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:53.647 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:43:53.647 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:43:53.647 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.632+0000 7fd63a930640 1 librados: starting msgr at 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.632+0000 7fd63a930640 1 librados: starting objecter 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.633+0000 7fd63a930640 1 Processor -- start 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.633+0000 7fd63a930640 1 -- start start 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.633+0000 7fd63a930640 1 librados: setting wanted keys 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.633+0000 7fd63a930640 1 librados: calling monclient init 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.633+0000 7fd63a930640 10 monclient: init 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd63a930640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:53.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd63a930640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd63a930640 10 monclient(hunting): picked mon.noname-a con 0x7fd6340a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd63a930640 10 monclient(hunting): picked mon.noname-c con 0x7fd63413f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd63a930640 10 monclient(hunting): picked mon.noname-b con 0x7fd6341457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd63a930640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd634137c50 con 0x7fd6340a9d70 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd63a930640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd634142d90 con 0x7fd6341457f0 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd63a930640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6340ac160 con 0x7fd63413f650 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd63a930640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd63a930640 10 monclient(hunting): authenticate will time out at 1415.761704s 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd633fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd6340a9d70 0x7fd634142680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34922/0 (socket says 172.21.15.155:34922) 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.634+0000 7fd633fff640 1 -- 172.21.15.155:0/1971361460 learned_addr learned my addr 172.21.15.155:0/1971361460 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 client.?.objecter ms_handle_connect 0x7fd63413f650 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 client.?.objecter ms_handle_connect 0x7fd6341457f0 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:53.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 client.?.objecter ms_handle_connect 0x7fd6340a9d70 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820694085 0 0) 0x7fd634142d90 con 0x7fd6341457f0 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd620004eb0 con 0x7fd6341457f0 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528281065 0 0) 0x7fd6340ac160 con 0x7fd63413f650 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.635+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd634142d90 con 0x7fd63413f650 2024-03-29T11:43:53.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356537524 0 0) 0x7fd634137c50 con 0x7fd6340a9d70 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6340ac160 con 0x7fd6340a9d70 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1093603345 0 0) 0x7fd620004eb0 con 0x7fd6341457f0 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd634137c50 con 0x7fd6341457f0 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd62c001bf0 con 0x7fd6341457f0 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.651 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3927569423 0 0) 0x7fd634142d90 con 0x7fd63413f650 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd620004eb0 con 0x7fd63413f650 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd628001b00 con 0x7fd63413f650 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:53.652 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.636+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1606900188 0 0) 0x7fd6340ac160 con 0x7fd6340a9d70 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd634142d90 con 0x7fd6340a9d70 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd624001910 con 0x7fd6340a9d70 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:53.653 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1525574822 0 0) 0x7fd634137c50 con 0x7fd6341457f0 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 >> v1:172.21.15.155:6790/0 conn(0x7fd63413f650 legacy=0x7fd634136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 >> v1:172.21.15.155:6789/0 conn(0x7fd6340a9d70 legacy=0x7fd634142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 1 monclient: found mon.b 2024-03-29T11:43:53.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6340ac440 con 0x7fd6341457f0 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd630ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd63a930640 5 monclient: authenticate success, global_id 4507 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd63a930640 10 monclient: _renew_subs 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd63a930640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd63a930640 1 -- 172.21.15.155:0/1971361460 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd634141d10 con 0x7fd6341457f0 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd63a930640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd63a930640 10 monclient: _renew_subs 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd63a930640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd63a930640 1 -- 172.21.15.155:0/1971361460 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63413fda0 con 0x7fd6341457f0 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.637+0000 7fd63a930640 1 librados: init done 2024-03-29T11:43:53.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.638+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd62c001d50 con 0x7fd6341457f0 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.638+0000 7fd630ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.638+0000 7fd63a930640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.638+0000 7fd6327fc640 4 set_mon_vals no callback set 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.639+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fd62c003b10 con 0x7fd6341457f0 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.639+0000 7fd63a930640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.639+0000 7fd63a930640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.639+0000 7fd63a930640 1 -- 172.21.15.155:0/1971361460 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd634075f10 con 0x7fd6341457f0 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.639+0000 7fd630ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.639+0000 7fd630ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.639+0000 7fd630ff9640 10 monclient: dump: 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:43:53.656 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.642+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fd62c019ae0 con 0x7fd6341457f0 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.643+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fd62c083af0 con 0x7fd6341457f0 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.643+0000 7fd630ff9640 10 client.4507.objecter ms_dispatch 0x7fd634000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.643+0000 7fd630ff9640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.643+0000 7fd633fff640 10 monclient: get_auth_request con 0x7fd62006e0f0 auth_method 0 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.643+0000 7fd630ff9640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.643+0000 7fd630ff9640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-29T11:43:53.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.643+0000 7fd630ff9640 10 client.4507.objecter ms_handle_connect 0x7fd62006e0f0 2024-03-29T11:43:53.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.649+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd62c04f200 con 0x7fd6341457f0 2024-03-29T11:43:53.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.649+0000 7fd630ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:53.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:53.649+0000 7fd630ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:54.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.000+0000 7fd63a930640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:54.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.000+0000 7fd63a930640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:54.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.000+0000 7fd63a930640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:43:54.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.000+0000 7fd63a930640 1 -- 172.21.15.155:0/1971361460 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6341422a0 con 0x7fd6341457f0 2024-03-29T11:43:54.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.001+0000 7fd630ff9640 1 -- 172.21.15.155:0/1971361460 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fd62c051300 con 0x7fd6341457f0 2024-03-29T11:43:54.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.001+0000 7fd630ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:54.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.001+0000 7fd630ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:54.007 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:43:54.007 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:54.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.005+0000 7fd63a930640 10 librados: watch_flush enter 2024-03-29T11:43:54.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.005+0000 7fd63a930640 10 librados: watch_flush exit 2024-03-29T11:43:54.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.005+0000 7fd63a930640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-29T11:43:54.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.005+0000 7fd63a930640 10 client.4507.objecter successfully canceled tick 2024-03-29T11:43:54.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.005+0000 7fd63a930640 1 -- 172.21.15.155:0/1971361460 >> v1:172.21.15.155:6816/1760814708 conn(0x7fd62006e0f0 legacy=0x7fd6200705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:54.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.005+0000 7fd63a930640 10 monclient: shutdown 2024-03-29T11:43:54.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.005+0000 7fd63a930640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:54.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.005+0000 7fd63a930640 1 -- 172.21.15.155:0/1971361460 >> v1:172.21.15.183:6789/0 conn(0x7fd6341457f0 legacy=0x7fd634137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:54.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.005+0000 7fd63a930640 1 -- 172.21.15.155:0/1971361460 shutdown_connections 2024-03-29T11:43:54.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.005+0000 7fd63a930640 1 -- 172.21.15.155:0/1971361460 >> 172.21.15.155:0/1971361460 conn(0x7fd634066a00 msgr2=0x7fd634066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:54.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.006+0000 7fd63a930640 1 -- 172.21.15.155:0/1971361460 shutdown_connections 2024-03-29T11:43:54.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.006+0000 7fd63a930640 1 -- 172.21.15.155:0/1971361460 wait complete. 2024-03-29T11:43:54.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:43:54.006+0000 7fd63a930640 1 librados: shutdown 2024-03-29T11:43:54.026 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:00.833 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:00.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.981+0000 7f665359e640 10 monclient: get_monmap_and_config 2024-03-29T11:44:00.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.982+0000 7f665359e640 10 monclient: build_initial_monmap 2024-03-29T11:44:00.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.982+0000 7f665359e640 10 monclient: monmap: 2024-03-29T11:44:00.987 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:00.987 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:00.987 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:44:00.983489+0000 2024-03-29T11:44:00.988 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:44:00.983489+0000 2024-03-29T11:44:00.988 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:00.988 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:00.988 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:00.988 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:44:00.988 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:44:00.988 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:00.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.984+0000 7f665359e640 1 Processor -- start 2024-03-29T11:44:00.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.984+0000 7f665359e640 1 -- start start 2024-03-29T11:44:00.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.984+0000 7f665359e640 10 monclient: init 2024-03-29T11:44:00.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.985+0000 7f665359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:00.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.985+0000 7f665359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:00.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.985+0000 7f665359e640 10 monclient(hunting): picked mon.noname-b con 0x7f664c034850 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:00.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.985+0000 7f665359e640 10 monclient(hunting): picked mon.noname-a con 0x7f664c02a640 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:00.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.985+0000 7f665359e640 10 monclient(hunting): picked mon.noname-c con 0x7f664c0327d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:00.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.985+0000 7f665359e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c02c820 con 0x7f664c02a640 2024-03-29T11:44:00.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.985+0000 7f665359e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c02a120 con 0x7f664c0327d0 2024-03-29T11:44:00.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.985+0000 7f665359e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0c75b0 con 0x7f664c034850 2024-03-29T11:44:00.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.985+0000 7f665359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:00.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.985+0000 7f665359e640 10 monclient(hunting): authenticate will time out at 1423.112429s 2024-03-29T11:44:00.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.989+0000 7f6651d9b640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f664c02a640 0x7f664c0320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:59993/0 (socket says 172.21.15.155:59993) 2024-03-29T11:44:00.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.989+0000 7f6651d9b640 1 -- 172.21.15.155:0/741018702 learned_addr learned my addr 172.21.15.155:0/741018702 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:00.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190187892 0 0) 0x7f664c02c820 con 0x7f664c02a640 2024-03-29T11:44:00.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:00.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:00.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663c003520 con 0x7f664c02a640 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632089172 0 0) 0x7f664c02a120 con 0x7f664c0327d0 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.993+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f664c02c820 con 0x7f664c0327d0 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171723002 0 0) 0x7f664c0c75b0 con 0x7f664c034850 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:44:00.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f664c02a120 con 0x7f664c034850 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 479722910 0 0) 0x7f663c003520 con 0x7f664c02a640 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 >> v1:172.21.15.155:6790/0 conn(0x7f664c034850 legacy=0x7f664c0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 >> v1:172.21.15.183:6789/0 conn(0x7f664c0327d0 legacy=0x7f664c0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient: _finish_hunting 0 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 1 monclient: found mon.noname-a 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c033110 con 0x7f664c02a640 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient: _finish_auth 0 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient: _check_auth_tickets 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665359e640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:01.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f665404f730 con 0x7f664c02a640 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient: dump: 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:01.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66540008d0 con 0x7f664c02a640 2024-03-29T11:44:01.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:01.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.994+0000 7f665359e640 4 set_mon_vals no callback set 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.995+0000 7f665159a640 1 -- 172.21.15.155:0/741018702 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f66540758b0 con 0x7f664c02a640 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.995+0000 7f665159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.995+0000 7f665159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.995+0000 7f665159a640 10 monclient: dump: 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:01.003 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:01.004 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:01.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.995+0000 7f665359e640 10 monclient: shutdown 2024-03-29T11:44:01.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.995+0000 7f665359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:01.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.995+0000 7f665359e640 1 -- 172.21.15.155:0/741018702 >> v1:172.21.15.155:6789/0 conn(0x7f664c02a640 legacy=0x7f664c0320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.996+0000 7f665359e640 1 -- 172.21.15.155:0/741018702 shutdown_connections 2024-03-29T11:44:01.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.996+0000 7f665359e640 1 -- 172.21.15.155:0/741018702 >> 172.21.15.155:0/741018702 conn(0x7f664c0058f0 msgr2=0x7f664c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:01.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.998+0000 7f665359e640 1 -- 172.21.15.155:0/741018702 shutdown_connections 2024-03-29T11:44:01.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.998+0000 7f665359e640 1 -- 172.21.15.155:0/741018702 wait complete. 2024-03-29T11:44:01.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.999+0000 7f665359e640 10 monclient: build_initial_monmap 2024-03-29T11:44:01.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.999+0000 7f665359e640 10 monclient: monmap: 2024-03-29T11:44:01.006 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:01.006 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:01.006 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:44:01.006 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:44:01.006 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:01.006 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:01.006 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.999+0000 7f665359e640 1 librados: starting msgr at 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.999+0000 7f665359e640 1 librados: starting objecter 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.999+0000 7f665359e640 1 Processor -- start 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.999+0000 7f665359e640 1 -- start start 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.999+0000 7f665359e640 1 librados: setting wanted keys 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.999+0000 7f665359e640 1 librados: calling monclient init 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:00.999+0000 7f665359e640 10 monclient: init 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.000+0000 7f665359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.000+0000 7f665359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.000+0000 7f665359e640 10 monclient(hunting): picked mon.noname-a con 0x7f664c02a640 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:01.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.000+0000 7f665359e640 10 monclient(hunting): picked mon.noname-b con 0x7f664c0327d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:01.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.000+0000 7f665359e640 10 monclient(hunting): picked mon.noname-c con 0x7f664c034850 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:01.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.000+0000 7f665359e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0c6d20 con 0x7f664c02a640 2024-03-29T11:44:01.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.000+0000 7f665359e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c02ec80 con 0x7f664c0327d0 2024-03-29T11:44:01.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.000+0000 7f665359e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c02ee10 con 0x7f664c034850 2024-03-29T11:44:01.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.000+0000 7f665359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:01.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.000+0000 7f665359e640 10 monclient(hunting): authenticate will time out at 1423.127674s 2024-03-29T11:44:01.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.007+0000 7f6651d9b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f664c0327d0 0x7f664c0bcfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:49389/0 (socket says 172.21.15.155:49389) 2024-03-29T11:44:01.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.007+0000 7f6651d9b640 1 -- 172.21.15.155:0/2741370581 learned_addr learned my addr 172.21.15.155:0/2741370581 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:01.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.011+0000 7f6642ffd640 10 client.?.objecter ms_handle_connect 0x7f664c0327d0 2024-03-29T11:44:01.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.011+0000 7f6642ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:01.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.011+0000 7f6642ffd640 10 client.?.objecter ms_handle_connect 0x7f664c034850 2024-03-29T11:44:01.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.011+0000 7f6642ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.011+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648279289 0 0) 0x7f664c02ee10 con 0x7f664c034850 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.011+0000 7f6642ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.011+0000 7f6642ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.011+0000 7f6642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.011+0000 7f6642ffd640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.011+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6634004eb0 con 0x7f664c034850 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1292702925 0 0) 0x7f6634004eb0 con 0x7f664c034850 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f664c02ee10 con 0x7f664c034850 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6644001780 con 0x7f664c034850 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:01.019 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130457928 0 0) 0x7f664c02ec80 con 0x7f664c0327d0 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:01.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.012+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6634004eb0 con 0x7f664c0327d0 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.013+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 768903646 0 0) 0x7f6634004eb0 con 0x7f664c0327d0 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.013+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f664c02ec80 con 0x7f664c0327d0 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.013+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f665404f400 con 0x7f664c0327d0 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.013+0000 7f6642ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.013+0000 7f6642ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.013+0000 7f6642ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:01.024 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2065614264 0 0) 0x7f664c02ee10 con 0x7f664c034850 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 >> v1:172.21.15.183:6789/0 conn(0x7f664c0327d0 legacy=0x7f664c0bcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 >> v1:172.21.15.155:6789/0 conn(0x7f664c02a640 legacy=0x7f664c02e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f6642ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f6642ffd640 1 monclient: found mon.c 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f6642ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f6642ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c0bd6b0 con 0x7f664c034850 2024-03-29T11:44:01.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f6642ffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f6642ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f6642ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f665359e640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f665359e640 10 monclient: _renew_subs 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f665359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f665359e640 1 -- 172.21.15.155:0/2741370581 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c031a40 con 0x7f664c034850 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f665359e640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f665359e640 10 monclient: _renew_subs 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f665359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f665359e640 1 -- 172.21.15.155:0/2741370581 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c030590 con 0x7f664c034850 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.017+0000 7f665359e640 1 librados: init done 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.018+0000 7f6640ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.018+0000 7f6640ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:01.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.018+0000 7f6640ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.018+0000 7f6640ff9640 1 -- 172.21.15.155:0/2741370581 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c0bd920 con 0x7f664c034850 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.019+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66440018e0 con 0x7f664c034850 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.019+0000 7f6642ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.019+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f66440037b0 con 0x7f664c034850 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.019+0000 7f6642ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.019+0000 7f6642ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.019+0000 7f6642ffd640 10 monclient: dump: 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:01.027 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:01.028 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:01.028 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:01.028 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:01.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.021+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f6644019780 con 0x7f664c034850 2024-03-29T11:44:01.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.022+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f6644083760 con 0x7f664c034850 2024-03-29T11:44:01.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.022+0000 7f6642ffd640 10 client.4622.objecter ms_dispatch 0x7f6654000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:01.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.022+0000 7f6642ffd640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:01.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.022+0000 7f6642ffd640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:01.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.022+0000 7f6642ffd640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:44:01.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.022+0000 7f6643fff640 4 set_mon_vals no callback set 2024-03-29T11:44:01.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.022+0000 7f665259c640 10 monclient: get_auth_request con 0x7f663406de60 auth_method 0 2024-03-29T11:44:01.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.023+0000 7f6642ffd640 10 client.4622.objecter ms_handle_connect 0x7f663406de60 2024-03-29T11:44:01.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.029+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f664404f040 con 0x7f664c034850 2024-03-29T11:44:01.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.029+0000 7f6642ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:01.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.029+0000 7f6642ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:01.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.383+0000 7f6640ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:01.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.383+0000 7f6640ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:01.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.383+0000 7f6640ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:01.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.383+0000 7f6640ff9640 1 -- 172.21.15.155:0/2741370581 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f664c004cd0 con 0x7f664c034850 2024-03-29T11:44:01.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.383+0000 7f6642ffd640 1 -- 172.21.15.155:0/2741370581 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f6644050f70 con 0x7f664c034850 2024-03-29T11:44:01.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.383+0000 7f6642ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:01.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.383+0000 7f6642ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:01.388 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:44:01.388 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:01.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 10 librados: watch_flush enter 2024-03-29T11:44:01.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 10 librados: watch_flush exit 2024-03-29T11:44:01.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:44:01.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:44:01.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 1 -- 172.21.15.155:0/2741370581 >> v1:172.21.15.155:6816/1760814708 conn(0x7f663406de60 legacy=0x7f6634070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 10 monclient: shutdown 2024-03-29T11:44:01.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:01.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 1 -- 172.21.15.155:0/2741370581 >> v1:172.21.15.155:6790/0 conn(0x7f664c034850 legacy=0x7f664c16c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 1 -- 172.21.15.155:0/2741370581 shutdown_connections 2024-03-29T11:44:01.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 1 -- 172.21.15.155:0/2741370581 >> 172.21.15.155:0/2741370581 conn(0x7f664c0058f0 msgr2=0x7f664c0c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:01.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 1 -- 172.21.15.155:0/2741370581 shutdown_connections 2024-03-29T11:44:01.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.387+0000 7f6640ff9640 1 -- 172.21.15.155:0/2741370581 wait complete. 2024-03-29T11:44:01.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:01.388+0000 7f6640ff9640 1 librados: shutdown 2024-03-29T11:44:01.407 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:08.213 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:08.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.376+0000 7fede1d9d640 10 monclient: get_monmap_and_config 2024-03-29T11:44:08.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.376+0000 7fede1d9d640 10 monclient: build_initial_monmap 2024-03-29T11:44:08.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.376+0000 7fede1d9d640 10 monclient: monmap: 2024-03-29T11:44:08.380 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:08.380 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:08.380 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:44:08.377178+0000 2024-03-29T11:44:08.381 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:44:08.377178+0000 2024-03-29T11:44:08.381 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:08.381 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:08.381 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:08.381 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:44:08.381 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:44:08.381 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:08.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.378+0000 7fede1d9d640 1 Processor -- start 2024-03-29T11:44:08.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.379+0000 7fede1d9d640 1 -- start start 2024-03-29T11:44:08.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.379+0000 7fede1d9d640 10 monclient: init 2024-03-29T11:44:08.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.380+0000 7fede1d9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:08.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.380+0000 7fede1d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:08.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.380+0000 7fede1d9d640 10 monclient(hunting): picked mon.noname-a con 0x7feddc0b1f60 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:08.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.380+0000 7fede1d9d640 10 monclient(hunting): picked mon.noname-b con 0x7feddc071eb0 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:08.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.380+0000 7fede1d9d640 10 monclient(hunting): picked mon.noname-c con 0x7feddc143d10 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:08.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.380+0000 7fede1d9d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc06ef80 con 0x7feddc0b1f60 2024-03-29T11:44:08.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.380+0000 7fede1d9d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc06f260 con 0x7feddc143d10 2024-03-29T11:44:08.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.380+0000 7fede1d9d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc0b40c0 con 0x7feddc071eb0 2024-03-29T11:44:08.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.380+0000 7fede1d9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:08.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.380+0000 7fede1d9d640 10 monclient(hunting): authenticate will time out at 1430.506376s 2024-03-29T11:44:08.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.383+0000 7feddbfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7feddc143d10 0x7feddc146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:36383/0 (socket says 172.21.15.155:36383) 2024-03-29T11:44:08.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.383+0000 7feddbfff640 1 -- 172.21.15.155:0/3923962557 learned_addr learned my addr 172.21.15.155:0/3923962557 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:08.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.384+0000 7fedda7fc640 1 -- 172.21.15.155:0/3923962557 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026599511 0 0) 0x7feddc06f260 con 0x7feddc143d10 2024-03-29T11:44:08.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.384+0000 7fedda7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:08.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.384+0000 7fedda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:08.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.384+0000 7fedda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:08.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.384+0000 7fedda7fc640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:44:08.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.384+0000 7fedda7fc640 1 -- 172.21.15.155:0/3923962557 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc4003520 con 0x7feddc143d10 2024-03-29T11:44:08.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 1 -- 172.21.15.155:0/3923962557 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2051383360 0 0) 0x7fedc4003520 con 0x7feddc143d10 2024-03-29T11:44:08.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 1 -- 172.21.15.155:0/3923962557 >> v1:172.21.15.155:6790/0 conn(0x7feddc071eb0 legacy=0x7feddc146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:08.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 1 -- 172.21.15.155:0/3923962557 >> v1:172.21.15.155:6789/0 conn(0x7feddc0b1f60 legacy=0x7feddc143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:08.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:08.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 1 monclient: found mon.noname-c 2024-03-29T11:44:08.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 1 -- 172.21.15.155:0/3923962557 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc155d70 con 0x7feddc143d10 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 1 -- 172.21.15.155:0/3923962557 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fedd4001a10 con 0x7feddc143d10 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 10 monclient: dump: 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:08.389 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:08.390 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:08.390 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:08.390 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:08.390 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:08.390 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:08.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fede1d9d640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:44:08.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fede1d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:08.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 1 -- 172.21.15.155:0/3923962557 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedd4001be0 con 0x7feddc143d10 2024-03-29T11:44:08.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fedda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:08.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fede1d9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:08.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.385+0000 7fede1d9d640 4 set_mon_vals no callback set 2024-03-29T11:44:08.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.386+0000 7fede1d9d640 10 monclient: shutdown 2024-03-29T11:44:08.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.386+0000 7fede1d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:08.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.386+0000 7fede1d9d640 1 -- 172.21.15.155:0/3923962557 >> v1:172.21.15.183:6789/0 conn(0x7feddc143d10 legacy=0x7feddc146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:08.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.386+0000 7fede1d9d640 1 -- 172.21.15.155:0/3923962557 shutdown_connections 2024-03-29T11:44:08.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.386+0000 7fede1d9d640 1 -- 172.21.15.155:0/3923962557 >> 172.21.15.155:0/3923962557 conn(0x7feddc066a00 msgr2=0x7feddc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:08.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.388+0000 7fede1d9d640 1 -- 172.21.15.155:0/3923962557 shutdown_connections 2024-03-29T11:44:08.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.388+0000 7fede1d9d640 1 -- 172.21.15.155:0/3923962557 wait complete. 2024-03-29T11:44:08.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.389+0000 7fede1d9d640 10 monclient: build_initial_monmap 2024-03-29T11:44:08.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.389+0000 7fede1d9d640 10 monclient: monmap: 2024-03-29T11:44:08.397 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:08.397 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:08.397 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:44:08.397 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:44:08.397 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:08.397 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:08.397 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:08.397 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:44:08.397 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:44:08.397 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:08.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.390+0000 7fede1d9d640 1 librados: starting msgr at 2024-03-29T11:44:08.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.390+0000 7fede1d9d640 1 librados: starting objecter 2024-03-29T11:44:08.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.390+0000 7fede1d9d640 1 Processor -- start 2024-03-29T11:44:08.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.390+0000 7fede1d9d640 1 -- start start 2024-03-29T11:44:08.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.390+0000 7fede1d9d640 1 librados: setting wanted keys 2024-03-29T11:44:08.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.390+0000 7fede1d9d640 1 librados: calling monclient init 2024-03-29T11:44:08.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.390+0000 7fede1d9d640 10 monclient: init 2024-03-29T11:44:08.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.391+0000 7fede1d9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:08.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.391+0000 7fede1d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:08.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.391+0000 7fede1d9d640 10 monclient(hunting): picked mon.noname-a con 0x7feddc141c70 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:08.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.391+0000 7fede1d9d640 10 monclient(hunting): picked mon.noname-c con 0x7feddc142050 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:08.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.394+0000 7fede1d9d640 10 monclient(hunting): picked mon.noname-b con 0x7feddc06f890 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:08.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.394+0000 7fede1d9d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc070910 con 0x7feddc141c70 2024-03-29T11:44:08.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.394+0000 7fede1d9d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc070bf0 con 0x7feddc06f890 2024-03-29T11:44:08.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.394+0000 7fede1d9d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc070ed0 con 0x7feddc142050 2024-03-29T11:44:08.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.394+0000 7fede1d9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:08.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.394+0000 7fede1d9d640 10 monclient(hunting): authenticate will time out at 1430.520724s 2024-03-29T11:44:08.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.394+0000 7feddbfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7feddc06f890 0x7feddc1ef430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:36465/0 (socket says 172.21.15.155:36465) 2024-03-29T11:44:08.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.394+0000 7feddbfff640 1 -- 172.21.15.155:0/1223320971 learned_addr learned my addr 172.21.15.155:0/1223320971 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:08.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.395+0000 7fedc3fff640 10 client.?.objecter ms_handle_connect 0x7feddc06f890 2024-03-29T11:44:08.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.395+0000 7fedc3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.396+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82351113 0 0) 0x7feddc070bf0 con 0x7feddc06f890 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.396+0000 7fedc3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.396+0000 7fedc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.396+0000 7fedc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.396+0000 7fedc3fff640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.396+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc8004eb0 con 0x7feddc06f890 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.396+0000 7fedc3fff640 10 client.?.objecter ms_handle_connect 0x7feddc142050 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.396+0000 7fedc3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2153234931 0 0) 0x7fedc8004eb0 con 0x7feddc06f890 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feddc070bf0 con 0x7feddc06f890 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fedd4004b60 con 0x7feddc06f890 2024-03-29T11:44:08.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient(hunting): dump: 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:08.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003305046 0 0) 0x7feddc070ed0 con 0x7feddc142050 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc8005680 con 0x7feddc142050 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3705985807 0 0) 0x7feddc070bf0 con 0x7feddc06f890 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 >> v1:172.21.15.155:6790/0 conn(0x7feddc142050 legacy=0x7feddc1ebd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 >> v1:172.21.15.155:6789/0 conn(0x7feddc141c70 legacy=0x7feddc130810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 1 monclient: found mon.b 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc06fd40 con 0x7feddc06f890 2024-03-29T11:44:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient: _finish_auth 0 2024-03-29T11:44:08.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:08.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fedc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:08.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fede1d9d640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:44:08.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fede1d9d640 10 monclient: _renew_subs 2024-03-29T11:44:08.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fede1d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:08.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fede1d9d640 1 -- 172.21.15.155:0/1223320971 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feddc0712f0 con 0x7feddc06f890 2024-03-29T11:44:08.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fede1d9d640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:08.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fede1d9d640 10 monclient: _renew_subs 2024-03-29T11:44:08.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fede1d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:08.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fede1d9d640 1 -- 172.21.15.155:0/1223320971 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feddc131100 con 0x7feddc06f890 2024-03-29T11:44:08.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.397+0000 7fede1d9d640 1 librados: init done 2024-03-29T11:44:08.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.398+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedd4001e50 con 0x7feddc06f890 2024-03-29T11:44:08.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.398+0000 7fedc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:08.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.398+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fedd4004b60 con 0x7feddc06f890 2024-03-29T11:44:08.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.398+0000 7fedc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:08.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.398+0000 7fedc3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:08.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.398+0000 7fedd97fa640 4 set_mon_vals no callback set 2024-03-29T11:44:08.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.398+0000 7fedc3fff640 10 monclient: dump: 2024-03-29T11:44:08.406 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:08.406 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:08.406 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:08.406 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.399+0000 7fede1d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.399+0000 7fede1d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.399+0000 7fede1d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.399+0000 7fede1d9d640 1 -- 172.21.15.155:0/1223320971 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedbc005080 con 0x7feddc06f890 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.399+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fedd4005030 con 0x7feddc06f890 2024-03-29T11:44:08.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.400+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7fedd4084ab0 con 0x7feddc06f890 2024-03-29T11:44:08.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.400+0000 7fedc3fff640 10 client.4519.objecter ms_dispatch 0x7feddc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:08.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.400+0000 7fedc3fff640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:08.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.400+0000 7fedc3fff640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:08.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.400+0000 7fedc3fff640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-29T11:44:08.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.401+0000 7feddb7fe640 10 monclient: get_auth_request con 0x7fedc806dff0 auth_method 0 2024-03-29T11:44:08.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.401+0000 7fedc3fff640 10 client.4519.objecter ms_handle_connect 0x7fedc806dff0 2024-03-29T11:44:08.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.407+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fedd4050390 con 0x7feddc06f890 2024-03-29T11:44:08.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.407+0000 7fedc3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:08.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.407+0000 7fedc3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:08.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.742+0000 7fede1d9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:08.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.742+0000 7fede1d9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:08.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.742+0000 7fede1d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:08.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.742+0000 7fede1d9d640 1 -- 172.21.15.155:0/1223320971 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedbc005330 con 0x7feddc06f890 2024-03-29T11:44:08.747 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:44:08.747 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:08.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.743+0000 7fedc3fff640 1 -- 172.21.15.155:0/1223320971 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7fedd4050390 con 0x7feddc06f890 2024-03-29T11:44:08.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.743+0000 7fedc3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:08.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.743+0000 7fedc3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:08.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 10 librados: watch_flush enter 2024-03-29T11:44:08.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 10 librados: watch_flush exit 2024-03-29T11:44:08.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-29T11:44:08.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 10 client.4519.objecter successfully canceled tick 2024-03-29T11:44:08.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 1 -- 172.21.15.155:0/1223320971 >> v1:172.21.15.155:6816/1760814708 conn(0x7fedc806dff0 legacy=0x7fedc80704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:08.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 10 monclient: shutdown 2024-03-29T11:44:08.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:08.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 1 -- 172.21.15.155:0/1223320971 >> v1:172.21.15.183:6789/0 conn(0x7feddc06f890 legacy=0x7feddc1ef430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:08.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 1 -- 172.21.15.155:0/1223320971 shutdown_connections 2024-03-29T11:44:08.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 1 -- 172.21.15.155:0/1223320971 >> 172.21.15.155:0/1223320971 conn(0x7feddc066a00 msgr2=0x7feddc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:08.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 1 -- 172.21.15.155:0/1223320971 shutdown_connections 2024-03-29T11:44:08.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.747+0000 7fede1d9d640 1 -- 172.21.15.155:0/1223320971 wait complete. 2024-03-29T11:44:08.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:08.748+0000 7fede1d9d640 1 librados: shutdown 2024-03-29T11:44:08.770 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:15.577 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:15.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.729+0000 7f9d51f15640 10 monclient: get_monmap_and_config 2024-03-29T11:44:15.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.729+0000 7f9d51f15640 10 monclient: build_initial_monmap 2024-03-29T11:44:15.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.729+0000 7f9d51f15640 10 monclient: monmap: 2024-03-29T11:44:15.733 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:15.733 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:15.733 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:44:15.729655+0000 2024-03-29T11:44:15.733 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:44:15.729655+0000 2024-03-29T11:44:15.733 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:15.733 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:15.733 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:15.733 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:44:15.733 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:44:15.734 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:15.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.730+0000 7f9d51f15640 1 Processor -- start 2024-03-29T11:44:15.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.731+0000 7f9d51f15640 1 -- start start 2024-03-29T11:44:15.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.731+0000 7f9d51f15640 10 monclient: init 2024-03-29T11:44:15.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.731+0000 7f9d51f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:15.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.731+0000 7f9d51f15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:15.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.731+0000 7f9d51f15640 10 monclient(hunting): picked mon.noname-a con 0x7f9d4c0b0860 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:15.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.732+0000 7f9d51f15640 10 monclient(hunting): picked mon.noname-b con 0x7f9d4c13ef30 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:15.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.732+0000 7f9d51f15640 10 monclient(hunting): picked mon.noname-c con 0x7f9d4c0af8e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:15.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.732+0000 7f9d51f15640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c0b0c40 con 0x7f9d4c0b0860 2024-03-29T11:44:15.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.732+0000 7f9d51f15640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c0aed10 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.732+0000 7f9d51f15640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c0afd50 con 0x7f9d4c13ef30 2024-03-29T11:44:15.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.732+0000 7f9d51f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:15.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.732+0000 7f9d51f15640 10 monclient(hunting): authenticate will time out at 1437.857927s 2024-03-29T11:44:15.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.734+0000 7f9d4affd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f9d4c13ef30 0x7f9d4c0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51312/0 (socket says 172.21.15.155:51312) 2024-03-29T11:44:15.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.734+0000 7f9d4affd640 1 -- 172.21.15.155:0/2516433538 learned_addr learned my addr 172.21.15.155:0/2516433538 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:15.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.735+0000 7f9d4a7fc640 1 -- 172.21.15.155:0/2516433538 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231514955 0 0) 0x7f9d4c0afd50 con 0x7f9d4c13ef30 2024-03-29T11:44:15.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.735+0000 7f9d4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:15.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.735+0000 7f9d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:15.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.735+0000 7f9d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:15.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.735+0000 7f9d4a7fc640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:44:15.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.735+0000 7f9d4a7fc640 1 -- 172.21.15.155:0/2516433538 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d34003520 con 0x7f9d4c13ef30 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 1 -- 172.21.15.155:0/2516433538 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44730848 0 0) 0x7f9d4c0aed10 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 1 -- 172.21.15.155:0/2516433538 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d4c0afd50 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 1 -- 172.21.15.155:0/2516433538 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2586969746 0 0) 0x7f9d34003520 con 0x7f9d4c13ef30 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 1 -- 172.21.15.155:0/2516433538 >> v1:172.21.15.183:6789/0 conn(0x7f9d4c0af8e0 legacy=0x7f9d4c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 1 -- 172.21.15.155:0/2516433538 >> v1:172.21.15.155:6789/0 conn(0x7f9d4c0b0860 legacy=0x7f9d4c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 1 -- 172.21.15.155:0/2516433538 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c0ab1b0 con 0x7f9d4c13ef30 2024-03-29T11:44:15.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:15.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:15.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:15.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 1 -- 172.21.15.155:0/2516433538 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f9d44001a10 con 0x7f9d4c13ef30 2024-03-29T11:44:15.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:15.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:15.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient: dump: 2024-03-29T11:44:15.741 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:15.741 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:15.741 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:15.742 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:15.742 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:15.742 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:15.742 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:15.742 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:15.742 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:15.742 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 1 -- 172.21.15.155:0/2516433538 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d44001b70 con 0x7f9d4c13ef30 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 1 -- 172.21.15.155:0/2516433538 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f9d44001f80 con 0x7f9d4c13ef30 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d4a7fc640 10 monclient: dump: 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:15.743 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:15.744 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:15.744 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:15.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d51f15640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:44:15.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d51f15640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:15.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.736+0000 7f9d51f15640 4 set_mon_vals no callback set 2024-03-29T11:44:15.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.737+0000 7f9d51f15640 10 monclient: shutdown 2024-03-29T11:44:15.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.737+0000 7f9d51f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:15.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.737+0000 7f9d51f15640 1 -- 172.21.15.155:0/2516433538 >> v1:172.21.15.155:6790/0 conn(0x7f9d4c13ef30 legacy=0x7f9d4c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:15.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.741+0000 7f9d51f15640 1 -- 172.21.15.155:0/2516433538 shutdown_connections 2024-03-29T11:44:15.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.741+0000 7f9d51f15640 1 -- 172.21.15.155:0/2516433538 >> 172.21.15.155:0/2516433538 conn(0x7f9d4c06b660 msgr2=0x7f9d4c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:15.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.743+0000 7f9d51f15640 1 -- 172.21.15.155:0/2516433538 shutdown_connections 2024-03-29T11:44:15.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.743+0000 7f9d51f15640 1 -- 172.21.15.155:0/2516433538 wait complete. 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.744+0000 7f9d51f15640 10 monclient: build_initial_monmap 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.744+0000 7f9d51f15640 10 monclient: monmap: 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:44:15.748 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:15.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.745+0000 7f9d51f15640 1 librados: starting msgr at 2024-03-29T11:44:15.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.745+0000 7f9d51f15640 1 librados: starting objecter 2024-03-29T11:44:15.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.745+0000 7f9d51f15640 1 Processor -- start 2024-03-29T11:44:15.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.745+0000 7f9d51f15640 1 -- start start 2024-03-29T11:44:15.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.745+0000 7f9d51f15640 1 librados: setting wanted keys 2024-03-29T11:44:15.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.745+0000 7f9d51f15640 1 librados: calling monclient init 2024-03-29T11:44:15.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.745+0000 7f9d51f15640 10 monclient: init 2024-03-29T11:44:15.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.746+0000 7f9d51f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:15.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.746+0000 7f9d51f15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:15.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.748+0000 7f9d51f15640 10 monclient(hunting): picked mon.noname-b con 0x7f9d4c0af8e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:15.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.748+0000 7f9d51f15640 10 monclient(hunting): picked mon.noname-a con 0x7f9d4c0b0860 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:15.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.748+0000 7f9d51f15640 10 monclient(hunting): picked mon.noname-c con 0x7f9d4c13ef30 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:15.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.749+0000 7f9d51f15640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c071be0 con 0x7f9d4c0b0860 2024-03-29T11:44:15.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.749+0000 7f9d51f15640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c1db770 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.749+0000 7f9d51f15640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c134a50 con 0x7f9d4c13ef30 2024-03-29T11:44:15.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.749+0000 7f9d51f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:15.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.749+0000 7f9d51f15640 10 monclient(hunting): authenticate will time out at 1437.875631s 2024-03-29T11:44:15.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.749+0000 7f9d4bfff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f9d4c13ef30 0x7f9d4c1d8920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:52246/0 (socket says 172.21.15.155:52246) 2024-03-29T11:44:15.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.749+0000 7f9d4bfff640 1 -- 172.21.15.155:0/1469743997 learned_addr learned my addr 172.21.15.155:0/1469743997 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:15.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 client.?.objecter ms_handle_connect 0x7f9d4c13ef30 2024-03-29T11:44:15.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:15.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 client.?.objecter ms_handle_connect 0x7f9d4c0af8e0 2024-03-29T11:44:15.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:15.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804411951 0 0) 0x7f9d4c134a50 con 0x7f9d4c13ef30 2024-03-29T11:44:15.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:15.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:15.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:15.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:44:15.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d38004eb0 con 0x7f9d4c13ef30 2024-03-29T11:44:15.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376222817 0 0) 0x7f9d4c1db770 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:15.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:15.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:15.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.750+0000 7f9d33fff640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:44:15.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.751+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d4c134a50 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 401320772 0 0) 0x7f9d4c134a50 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d4c1db770 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f9d3c001510 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:15.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 10 monclient(hunting): dump: 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 297104009 0 0) 0x7f9d4c1db770 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 >> v1:172.21.15.155:6790/0 conn(0x7f9d4c13ef30 legacy=0x7f9d4c1d8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:15.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 >> v1:172.21.15.155:6789/0 conn(0x7f9d4c0b0860 legacy=0x7f9d4c1dbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 1 monclient: found mon.b 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.752+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c06e6e0 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.753+0000 7f9d33fff640 10 monclient: _finish_auth 0 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.753+0000 7f9d33fff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.753+0000 7f9d33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.753+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d3c001670 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.753+0000 7f9d33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.753+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f9d3c001d40 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.753+0000 7f9d33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.753+0000 7f9d33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:15.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.753+0000 7f9d33fff640 10 monclient: dump: 2024-03-29T11:44:15.761 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:15.761 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:15.761 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:15.761 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:15.761 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:15.761 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:15.761 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:15.761 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:15.761 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:15.761 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:15.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.755+0000 7f9d51f15640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:44:15.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.755+0000 7f9d51f15640 10 monclient: _renew_subs 2024-03-29T11:44:15.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.755+0000 7f9d51f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:15.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.755+0000 7f9d51f15640 1 -- 172.21.15.155:0/1469743997 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4c1dc3a0 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.755+0000 7f9d51f15640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:15.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.755+0000 7f9d51f15640 10 monclient: _renew_subs 2024-03-29T11:44:15.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.755+0000 7f9d51f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:15.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.755+0000 7f9d49ffb640 4 set_mon_vals no callback set 2024-03-29T11:44:15.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.755+0000 7f9d51f15640 1 -- 172.21.15.155:0/1469743997 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4c134190 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.755+0000 7f9d51f15640 1 librados: init done 2024-03-29T11:44:15.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.757+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f9d3c019700 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.758+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f9d3c083790 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.758+0000 7f9d33fff640 10 client.4525.objecter ms_dispatch 0x7f9d4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:15.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.758+0000 7f9d51f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:15.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.758+0000 7f9d51f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:15.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.758+0000 7f9d51f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:15.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.758+0000 7f9d51f15640 1 -- 172.21.15.155:0/1469743997 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4c06e870 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.758+0000 7f9d33fff640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:15.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.758+0000 7f9d33fff640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:15.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.758+0000 7f9d33fff640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-29T11:44:15.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.760+0000 7f9d4affd640 10 monclient: get_auth_request con 0x7f9d3806dd70 auth_method 0 2024-03-29T11:44:15.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.760+0000 7f9d33fff640 10 client.4525.objecter ms_handle_connect 0x7f9d3806dd70 2024-03-29T11:44:15.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.768+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d4c06e870 con 0x7f9d4c0af8e0 2024-03-29T11:44:15.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.768+0000 7f9d33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:15.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:15.768+0000 7f9d33fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:16.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.113+0000 7f9d51f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:16.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.113+0000 7f9d51f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:16.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.113+0000 7f9d51f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:16.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.113+0000 7f9d51f15640 1 -- 172.21.15.155:0/1469743997 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d4c06e870 con 0x7f9d4c0af8e0 2024-03-29T11:44:16.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.115+0000 7f9d33fff640 1 -- 172.21.15.155:0/1469743997 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f9d4c06e870 con 0x7f9d4c0af8e0 2024-03-29T11:44:16.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.115+0000 7f9d33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:16.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.115+0000 7f9d33fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:16.119 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:44:16.119 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:16.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.119+0000 7f9d51f15640 10 librados: watch_flush enter 2024-03-29T11:44:16.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.119+0000 7f9d51f15640 10 librados: watch_flush exit 2024-03-29T11:44:16.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.119+0000 7f9d51f15640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-29T11:44:16.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.119+0000 7f9d51f15640 10 client.4525.objecter successfully canceled tick 2024-03-29T11:44:16.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.119+0000 7f9d51f15640 1 -- 172.21.15.155:0/1469743997 >> v1:172.21.15.155:6816/1760814708 conn(0x7f9d3806dd70 legacy=0x7f9d38070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:16.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.119+0000 7f9d51f15640 10 monclient: shutdown 2024-03-29T11:44:16.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.119+0000 7f9d51f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:16.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.119+0000 7f9d51f15640 1 -- 172.21.15.155:0/1469743997 >> v1:172.21.15.183:6789/0 conn(0x7f9d4c0af8e0 legacy=0x7f9d4c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:16.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.120+0000 7f9d51f15640 1 -- 172.21.15.155:0/1469743997 shutdown_connections 2024-03-29T11:44:16.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.120+0000 7f9d51f15640 1 -- 172.21.15.155:0/1469743997 >> 172.21.15.155:0/1469743997 conn(0x7f9d4c06b660 msgr2=0x7f9d4c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:16.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.121+0000 7f9d51f15640 1 -- 172.21.15.155:0/1469743997 shutdown_connections 2024-03-29T11:44:16.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.121+0000 7f9d51f15640 1 -- 172.21.15.155:0/1469743997 wait complete. 2024-03-29T11:44:16.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:16.121+0000 7f9d51f15640 1 librados: shutdown 2024-03-29T11:44:16.142 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:22.950 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:23.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.097+0000 7f8a640fb640 10 monclient: get_monmap_and_config 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.097+0000 7f8a640fb640 10 monclient: build_initial_monmap 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.097+0000 7f8a640fb640 10 monclient: monmap: 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:44:23.097550+0000 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:44:23.097550+0000 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:44:23.101 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:23.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.097+0000 7f8a640fb640 1 Processor -- start 2024-03-29T11:44:23.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.098+0000 7f8a640fb640 1 -- start start 2024-03-29T11:44:23.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.098+0000 7f8a640fb640 10 monclient: init 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a640fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a640fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a640fb640 10 monclient(hunting): picked mon.noname-c con 0x7f8a5c1457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a640fb640 10 monclient(hunting): picked mon.noname-a con 0x7f8a5c13f650 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a640fb640 10 monclient(hunting): picked mon.noname-b con 0x7f8a5c0a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a640fb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c0aaef0 con 0x7f8a5c13f650 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a640fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c145bd0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a640fb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c145eb0 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a640fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a640fb640 10 monclient(hunting): authenticate will time out at 1445.225346s 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a61696640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f8a5c13f650 0x7f8a5c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:43520/0 (socket says 172.21.15.155:43520) 2024-03-29T11:44:23.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.099+0000 7f8a61696640 1 -- 172.21.15.155:0/1472183262 learned_addr learned my addr 172.21.15.155:0/1472183262 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:23.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900172747 0 0) 0x7f8a5c145eb0 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a48003520 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265746964 0 0) 0x7f8a5c0aaef0 con 0x7f8a5c13f650 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a5c145eb0 con 0x7f8a5c13f650 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918032314 0 0) 0x7f8a5c145bd0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.100+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a5c0aaef0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4007494390 0 0) 0x7f8a48003520 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 >> v1:172.21.15.183:6789/0 conn(0x7f8a5c1457f0 legacy=0x7f8a5c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 >> v1:172.21.15.155:6789/0 conn(0x7f8a5c13f650 legacy=0x7f8a5c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 10 monclient: _finish_hunting 0 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 1 monclient: found mon.noname-b 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c146340 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 10 monclient: _finish_auth 0 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 10 monclient: _check_auth_tickets 2024-03-29T11:44:23.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8a54001a10 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a60e95640 10 monclient: dump: 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:23.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a640fb640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:44:23.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.101+0000 7f8a640fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:23.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.102+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a54001c30 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.102+0000 7f8a60e95640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.102+0000 7f8a60e95640 1 -- 172.21.15.155:0/1472183262 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8a54001fc0 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.102+0000 7f8a60e95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.102+0000 7f8a60e95640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.102+0000 7f8a60e95640 10 monclient: dump: 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:23.109 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.102+0000 7f8a640fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.102+0000 7f8a640fb640 4 set_mon_vals no callback set 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.103+0000 7f8a640fb640 10 monclient: shutdown 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.103+0000 7f8a640fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.103+0000 7f8a640fb640 1 -- 172.21.15.155:0/1472183262 >> v1:172.21.15.155:6790/0 conn(0x7f8a5c0a9d70 legacy=0x7f8a5c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.103+0000 7f8a640fb640 1 -- 172.21.15.155:0/1472183262 shutdown_connections 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.103+0000 7f8a640fb640 1 -- 172.21.15.155:0/1472183262 >> 172.21.15.155:0/1472183262 conn(0x7f8a5c066a00 msgr2=0x7f8a5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.103+0000 7f8a640fb640 1 -- 172.21.15.155:0/1472183262 shutdown_connections 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.103+0000 7f8a640fb640 1 -- 172.21.15.155:0/1472183262 wait complete. 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.104+0000 7f8a640fb640 10 monclient: build_initial_monmap 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.104+0000 7f8a640fb640 10 monclient: monmap: 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:23.110 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:23.111 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:44:23.111 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:44:23.111 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:23.111 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:23.111 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:23.111 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:44:23.111 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:44:23.111 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:23.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.105+0000 7f8a640fb640 1 librados: starting msgr at 2024-03-29T11:44:23.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.105+0000 7f8a640fb640 1 librados: starting objecter 2024-03-29T11:44:23.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.105+0000 7f8a640fb640 1 Processor -- start 2024-03-29T11:44:23.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.105+0000 7f8a640fb640 1 -- start start 2024-03-29T11:44:23.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.105+0000 7f8a640fb640 1 librados: setting wanted keys 2024-03-29T11:44:23.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.105+0000 7f8a640fb640 1 librados: calling monclient init 2024-03-29T11:44:23.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.105+0000 7f8a640fb640 10 monclient: init 2024-03-29T11:44:23.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a640fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:23.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a640fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a640fb640 10 monclient(hunting): picked mon.noname-a con 0x7f8a5c0a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a640fb640 10 monclient(hunting): picked mon.noname-c con 0x7f8a5c13f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a640fb640 10 monclient(hunting): picked mon.noname-b con 0x7f8a5c1457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a640fb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c1cc960 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a640fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c1cd5b0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a640fb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c1ccbd0 con 0x7f8a5c13f650 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a640fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a640fb640 10 monclient(hunting): authenticate will time out at 1445.232554s 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a61696640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f8a5c13f650 0x7f8a5c1cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:41646/0 (socket says 172.21.15.155:41646) 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a61696640 1 -- 172.21.15.155:0/554136035 learned_addr learned my addr 172.21.15.155:0/554136035 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a527fc640 10 client.?.objecter ms_handle_connect 0x7f8a5c13f650 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.106+0000 7f8a527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 client.?.objecter ms_handle_connect 0x7f8a5c1457f0 2024-03-29T11:44:23.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 client.?.objecter ms_handle_connect 0x7f8a5c0a9d70 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165446002 0 0) 0x7f8a5c1cd5b0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a40004eb0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379478255 0 0) 0x7f8a5c1cc960 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:23.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a5c1cd5b0 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353964847 0 0) 0x7f8a5c1ccbd0 con 0x7f8a5c13f650 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a5c1cc960 con 0x7f8a5c13f650 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 800791813 0 0) 0x7f8a40004eb0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a5c1ccbd0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8a54001950 con 0x7f8a5c1457f0 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.107+0000 7f8a527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 10 monclient(hunting): dump: 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:23.117 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4097009042 0 0) 0x7f8a5c1cc960 con 0x7f8a5c13f650 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a40004eb0 con 0x7f8a5c13f650 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8a4c001930 con 0x7f8a5c13f650 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:23.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 10 monclient(hunting): dump: 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 541147210 0 0) 0x7f8a5c1cd5b0 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a40005df0 con 0x7f8a5c0a9d70 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4166601794 0 0) 0x7f8a5c1ccbd0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 >> v1:172.21.15.155:6790/0 conn(0x7f8a5c13f650 legacy=0x7f8a5c1cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 >> v1:172.21.15.155:6789/0 conn(0x7f8a5c0a9d70 legacy=0x7f8a5c13b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 1 monclient: found mon.b 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c1cceb0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a640fb640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a640fb640 10 monclient: _renew_subs 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a640fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a640fb640 1 -- 172.21.15.155:0/554136035 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c1cbe60 con 0x7f8a5c1457f0 2024-03-29T11:44:23.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a640fb640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a640fb640 10 monclient: _renew_subs 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a640fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a640fb640 1 -- 172.21.15.155:0/554136035 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c1e76e0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.108+0000 7f8a640fb640 1 librados: init done 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.109+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a54001590 con 0x7f8a5c1457f0 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.109+0000 7f8a527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.109+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8a54001ab0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.109+0000 7f8a527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.109+0000 7f8a527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.109+0000 7f8a527fc640 10 monclient: dump: 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:23.121 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.109+0000 7f8a53fff640 4 set_mon_vals no callback set 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.110+0000 7f8a640fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.110+0000 7f8a640fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.110+0000 7f8a640fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.110+0000 7f8a640fb640 1 -- 172.21.15.155:0/554136035 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a44005080 con 0x7f8a5c1457f0 2024-03-29T11:44:23.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.111+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f8a54019c30 con 0x7f8a5c1457f0 2024-03-29T11:44:23.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.111+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f8a54083af0 con 0x7f8a5c1457f0 2024-03-29T11:44:23.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.111+0000 7f8a527fc640 10 client.4531.objecter ms_dispatch 0x7f8a5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:23.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.111+0000 7f8a527fc640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:23.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.111+0000 7f8a527fc640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:23.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.111+0000 7f8a61e97640 10 monclient: get_auth_request con 0x7f8a4006e0f0 auth_method 0 2024-03-29T11:44:23.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.111+0000 7f8a527fc640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-29T11:44:23.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.112+0000 7f8a527fc640 10 client.4531.objecter ms_handle_connect 0x7f8a4006e0f0 2024-03-29T11:44:23.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.119+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a5404f200 con 0x7f8a5c1457f0 2024-03-29T11:44:23.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.119+0000 7f8a527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:23.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.119+0000 7f8a527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:23.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.472+0000 7f8a640fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:23.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.472+0000 7f8a640fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:23.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.472+0000 7f8a640fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:44:23.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.472+0000 7f8a640fb640 1 -- 172.21.15.155:0/554136035 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a44005330 con 0x7f8a5c1457f0 2024-03-29T11:44:23.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.474+0000 7f8a527fc640 1 -- 172.21.15.155:0/554136035 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f8a54051300 con 0x7f8a5c1457f0 2024-03-29T11:44:23.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.474+0000 7f8a527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:23.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.474+0000 7f8a527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:23.477 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:44:23.477 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:23.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.477+0000 7f8a640fb640 10 librados: watch_flush enter 2024-03-29T11:44:23.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.477+0000 7f8a640fb640 10 librados: watch_flush exit 2024-03-29T11:44:23.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.477+0000 7f8a640fb640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-29T11:44:23.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.477+0000 7f8a640fb640 10 client.4531.objecter successfully canceled tick 2024-03-29T11:44:23.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.478+0000 7f8a640fb640 1 -- 172.21.15.155:0/554136035 >> v1:172.21.15.155:6816/1760814708 conn(0x7f8a4006e0f0 legacy=0x7f8a400705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.478+0000 7f8a640fb640 10 monclient: shutdown 2024-03-29T11:44:23.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.478+0000 7f8a640fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:23.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.478+0000 7f8a640fb640 1 -- 172.21.15.155:0/554136035 >> v1:172.21.15.183:6789/0 conn(0x7f8a5c1457f0 legacy=0x7f8a5c1ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.478+0000 7f8a640fb640 1 -- 172.21.15.155:0/554136035 shutdown_connections 2024-03-29T11:44:23.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.478+0000 7f8a640fb640 1 -- 172.21.15.155:0/554136035 >> 172.21.15.155:0/554136035 conn(0x7f8a5c066a00 msgr2=0x7f8a5c13cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:23.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.478+0000 7f8a640fb640 1 -- 172.21.15.155:0/554136035 shutdown_connections 2024-03-29T11:44:23.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.478+0000 7f8a640fb640 1 -- 172.21.15.155:0/554136035 wait complete. 2024-03-29T11:44:23.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:23.478+0000 7f8a640fb640 1 librados: shutdown 2024-03-29T11:44:23.496 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:30.304 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:30.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.434+0000 7f873e02e640 10 monclient: get_monmap_and_config 2024-03-29T11:44:30.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.434+0000 7f873e02e640 10 monclient: build_initial_monmap 2024-03-29T11:44:30.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.434+0000 7f873e02e640 10 monclient: monmap: 2024-03-29T11:44:30.439 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:30.439 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:30.439 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:44:30.436181+0000 2024-03-29T11:44:30.439 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:44:30.436181+0000 2024-03-29T11:44:30.439 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:30.439 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:30.440 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:30.440 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:44:30.440 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:44:30.440 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:30.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.435+0000 7f873e02e640 1 Processor -- start 2024-03-29T11:44:30.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.436+0000 7f873e02e640 1 -- start start 2024-03-29T11:44:30.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.436+0000 7f873e02e640 10 monclient: init 2024-03-29T11:44:30.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f873e02e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:30.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f873e02e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:30.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f873e02e640 10 monclient(hunting): picked mon.noname-b con 0x7f87381422b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:30.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f873e02e640 10 monclient(hunting): picked mon.noname-c con 0x7f8738142690 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:30.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f873e02e640 10 monclient(hunting): picked mon.noname-a con 0x7f87380b17d0 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:30.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f873e02e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8738142c80 con 0x7f87380b17d0 2024-03-29T11:44:30.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f873e02e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87380aae40 con 0x7f8738142690 2024-03-29T11:44:30.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f873e02e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873813fbd0 con 0x7f87381422b0 2024-03-29T11:44:30.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f873e02e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:30.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f873e02e640 10 monclient(hunting): authenticate will time out at 1452.564385s 2024-03-29T11:44:30.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f87377fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f87381422b0 0x7f873813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:57108/0 (socket says 172.21.15.155:57108) 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.437+0000 7f87377fe640 1 -- 172.21.15.155:0/4041205215 learned_addr learned my addr 172.21.15.155:0/4041205215 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 1 -- 172.21.15.155:0/4041205215 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501595023 0 0) 0x7f873813fbd0 con 0x7f87381422b0 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 1 -- 172.21.15.155:0/4041205215 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8720003520 con 0x7f87381422b0 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 1 -- 172.21.15.155:0/4041205215 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222803611 0 0) 0x7f87380aae40 con 0x7f8738142690 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:44:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 1 -- 172.21.15.155:0/4041205215 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f873813fbd0 con 0x7f8738142690 2024-03-29T11:44:30.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 1 -- 172.21.15.155:0/4041205215 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1576309236 0 0) 0x7f8720003520 con 0x7f87381422b0 2024-03-29T11:44:30.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 1 -- 172.21.15.155:0/4041205215 >> v1:172.21.15.183:6789/0 conn(0x7f8738142690 legacy=0x7f87380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:30.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.438+0000 7f87367fc640 1 -- 172.21.15.155:0/4041205215 >> v1:172.21.15.155:6789/0 conn(0x7f87380b17d0 legacy=0x7f87380b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:30.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:30.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 1 monclient: found mon.noname-b 2024-03-29T11:44:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:44:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 1 -- 172.21.15.155:0/4041205215 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8738151940 con 0x7f87381422b0 2024-03-29T11:44:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 1 -- 172.21.15.155:0/4041205215 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8728001690 con 0x7f87381422b0 2024-03-29T11:44:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 10 monclient: dump: 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f873e02e640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:44:30.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f873e02e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 1 -- 172.21.15.155:0/4041205215 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87280017f0 con 0x7f87381422b0 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 1 -- 172.21.15.155:0/4041205215 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8728001bc0 con 0x7f87381422b0 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.439+0000 7f87367fc640 10 monclient: dump: 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:30.447 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:30.448 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:30.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.440+0000 7f873e02e640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:30.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.440+0000 7f873e02e640 4 set_mon_vals no callback set 2024-03-29T11:44:30.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.441+0000 7f873e02e640 10 monclient: shutdown 2024-03-29T11:44:30.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.441+0000 7f873e02e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:30.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.441+0000 7f873e02e640 1 -- 172.21.15.155:0/4041205215 >> v1:172.21.15.155:6790/0 conn(0x7f87381422b0 legacy=0x7f873813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:30.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.441+0000 7f873e02e640 1 -- 172.21.15.155:0/4041205215 shutdown_connections 2024-03-29T11:44:30.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.441+0000 7f873e02e640 1 -- 172.21.15.155:0/4041205215 >> 172.21.15.155:0/4041205215 conn(0x7f8738066a00 msgr2=0x7f8738066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:30.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.441+0000 7f873e02e640 1 -- 172.21.15.155:0/4041205215 shutdown_connections 2024-03-29T11:44:30.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.441+0000 7f873e02e640 1 -- 172.21.15.155:0/4041205215 wait complete. 2024-03-29T11:44:30.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.442+0000 7f873e02e640 10 monclient: build_initial_monmap 2024-03-29T11:44:30.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.442+0000 7f873e02e640 10 monclient: monmap: 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.443+0000 7f873e02e640 1 librados: starting msgr at 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.443+0000 7f873e02e640 1 librados: starting objecter 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.443+0000 7f873e02e640 1 Processor -- start 2024-03-29T11:44:30.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.443+0000 7f873e02e640 1 -- start start 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.443+0000 7f873e02e640 1 librados: setting wanted keys 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.443+0000 7f873e02e640 1 librados: calling monclient init 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.443+0000 7f873e02e640 10 monclient: init 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f873e02e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f873e02e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f873e02e640 10 monclient(hunting): picked mon.noname-c con 0x7f87380b17d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f873e02e640 10 monclient(hunting): picked mon.noname-a con 0x7f87381422b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f873e02e640 10 monclient(hunting): picked mon.noname-b con 0x7f8738142690 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f873e02e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873812f680 con 0x7f87381422b0 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f873e02e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87380acfd0 con 0x7f8738142690 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f873e02e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87380ad2b0 con 0x7f87380b17d0 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f873e02e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f873e02e640 10 monclient(hunting): authenticate will time out at 1452.571627s 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f87377fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f87380b17d0 0x7f87380ab540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:57632/0 (socket says 172.21.15.155:57632) 2024-03-29T11:44:30.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f87377fe640 1 -- 172.21.15.155:0/530613738 learned_addr learned my addr 172.21.15.155:0/530613738 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:30.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f871ffff640 10 client.?.objecter ms_handle_connect 0x7f87380b17d0 2024-03-29T11:44:30.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.444+0000 7f871ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:30.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903744189 0 0) 0x7f87380ad2b0 con 0x7f87380b17d0 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8724004eb0 con 0x7f87380b17d0 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 client.?.objecter ms_handle_connect 0x7f8738142690 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201507379 0 0) 0x7f87380acfd0 con 0x7f8738142690 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:30.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87380ad2b0 con 0x7f8738142690 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3280286491 0 0) 0x7f8724004eb0 con 0x7f87380b17d0 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8724005490 con 0x7f87380b17d0 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8728001bc0 con 0x7f87380b17d0 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): dump: 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:30.456 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.446+0000 7f871ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 104511485 0 0) 0x7f87380ad2b0 con 0x7f8738142690 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8724005ac0 con 0x7f8738142690 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8730001b10 con 0x7f8738142690 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 10 monclient(hunting): dump: 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:30.457 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1003677400 0 0) 0x7f8724005490 con 0x7f87380b17d0 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 >> v1:172.21.15.183:6789/0 conn(0x7f8738142690 legacy=0x7f873812ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 >> v1:172.21.15.155:6789/0 conn(0x7f87381422b0 legacy=0x7f873812e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 1 monclient: found mon.c 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8738151210 con 0x7f87380b17d0 2024-03-29T11:44:30.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 10 monclient: _finish_auth 0 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f873e02e640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f873e02e640 10 monclient: _renew_subs 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f873e02e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f873e02e640 1 -- 172.21.15.155:0/530613738 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873814f270 con 0x7f87380b17d0 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8728004a90 con 0x7f87380b17d0 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f8728001bc0 con 0x7f87380b17d0 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f873e02e640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f871ffff640 10 monclient: dump: 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:30.459 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f873e02e640 10 monclient: _renew_subs 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f873e02e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f873e02e640 1 -- 172.21.15.155:0/530613738 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87380ac250 con 0x7f87380b17d0 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f8735ffb640 4 set_mon_vals no callback set 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.447+0000 7f873e02e640 1 librados: init done 2024-03-29T11:44:30.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.449+0000 7f871dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:30.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.449+0000 7f871dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:30.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.449+0000 7f871dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:30.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.449+0000 7f871dffb640 1 -- 172.21.15.155:0/530613738 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8738075f10 con 0x7f87380b17d0 2024-03-29T11:44:30.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.449+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f872801aaf0 con 0x7f87380b17d0 2024-03-29T11:44:30.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.450+0000 7f8736ffd640 10 monclient: get_auth_request con 0x7f872406e690 auth_method 0 2024-03-29T11:44:30.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.450+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3127840454 0 0) 0x7f87280849b0 con 0x7f87380b17d0 2024-03-29T11:44:30.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.450+0000 7f871ffff640 10 client.4643.objecter ms_dispatch 0x7f8738000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:30.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.450+0000 7f871ffff640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:30.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.450+0000 7f871ffff640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:30.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.450+0000 7f871ffff640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T11:44:30.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.450+0000 7f871ffff640 10 client.4643.objecter ms_handle_connect 0x7f872406e690 2024-03-29T11:44:30.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.458+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87280503d0 con 0x7f87380b17d0 2024-03-29T11:44:30.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.458+0000 7f871ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:30.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.458+0000 7f871ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:30.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.758+0000 7f871dffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:30.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.758+0000 7f871dffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:30.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.758+0000 7f871dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:30.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.758+0000 7f871dffb640 1 -- 172.21.15.155:0/530613738 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87380b34b0 con 0x7f87380b17d0 2024-03-29T11:44:30.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.759+0000 7f871ffff640 1 -- 172.21.15.155:0/530613738 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 162942829) 0x7f87280503d0 con 0x7f87380b17d0 2024-03-29T11:44:30.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.759+0000 7f871ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:30.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.759+0000 7f871ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:30.764 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:44:30.764 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:34:33.906248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:33.906248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:30.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.763+0000 7f871dffb640 10 librados: watch_flush enter 2024-03-29T11:44:30.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.763+0000 7f871dffb640 10 librados: watch_flush exit 2024-03-29T11:44:30.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.763+0000 7f871dffb640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T11:44:30.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.763+0000 7f871dffb640 10 client.4643.objecter successfully canceled tick 2024-03-29T11:44:30.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.763+0000 7f871dffb640 1 -- 172.21.15.155:0/530613738 >> v1:172.21.15.155:6816/1760814708 conn(0x7f872406e690 legacy=0x7f8724070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:30.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.763+0000 7f871dffb640 10 monclient: shutdown 2024-03-29T11:44:30.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.763+0000 7f871dffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:30.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.763+0000 7f871dffb640 1 -- 172.21.15.155:0/530613738 >> v1:172.21.15.155:6790/0 conn(0x7f87380b17d0 legacy=0x7f87380ab540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:30.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.764+0000 7f871dffb640 1 -- 172.21.15.155:0/530613738 shutdown_connections 2024-03-29T11:44:30.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.764+0000 7f871dffb640 1 -- 172.21.15.155:0/530613738 >> 172.21.15.155:0/530613738 conn(0x7f8738066a00 msgr2=0x7f873813c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:30.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.764+0000 7f871dffb640 1 -- 172.21.15.155:0/530613738 shutdown_connections 2024-03-29T11:44:30.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.764+0000 7f871dffb640 1 -- 172.21.15.155:0/530613738 wait complete. 2024-03-29T11:44:30.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:30.764+0000 7f871dffb640 1 librados: shutdown 2024-03-29T11:44:30.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:37.591 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:37.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.779+0000 7fb266a81640 10 monclient: get_monmap_and_config 2024-03-29T11:44:37.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.779+0000 7fb266a81640 10 monclient: build_initial_monmap 2024-03-29T11:44:37.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.779+0000 7fb266a81640 10 monclient: monmap: 2024-03-29T11:44:37.785 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:37.785 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:37.785 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:44:37.781211+0000 2024-03-29T11:44:37.785 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:44:37.781211+0000 2024-03-29T11:44:37.785 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:37.785 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:37.785 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:37.785 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:44:37.785 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:44:37.785 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:37.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.780+0000 7fb266a81640 1 Processor -- start 2024-03-29T11:44:37.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.780+0000 7fb266a81640 1 -- start start 2024-03-29T11:44:37.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.780+0000 7fb266a81640 10 monclient: init 2024-03-29T11:44:37.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.781+0000 7fb266a81640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:37.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.781+0000 7fb266a81640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:37.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.781+0000 7fb266a81640 10 monclient(hunting): picked mon.noname-a con 0x7fb2601457f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:37.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.781+0000 7fb266a81640 10 monclient(hunting): picked mon.noname-b con 0x7fb26013f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:37.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.781+0000 7fb266a81640 10 monclient(hunting): picked mon.noname-c con 0x7fb2600a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:37.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.781+0000 7fb266a81640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2600aaef0 con 0x7fb2601457f0 2024-03-29T11:44:37.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.781+0000 7fb266a81640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb260145bd0 con 0x7fb2600a9d70 2024-03-29T11:44:37.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.781+0000 7fb266a81640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb260145eb0 con 0x7fb26013f650 2024-03-29T11:44:37.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.781+0000 7fb266a81640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:37.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.781+0000 7fb266a81640 10 monclient(hunting): authenticate will time out at 1459.908800s 2024-03-29T11:44:37.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.787+0000 7fb25ffff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fb26013f650 0x7fb2600b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:45228/0 (socket says 172.21.15.155:45228) 2024-03-29T11:44:37.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.787+0000 7fb25ffff640 1 -- 172.21.15.155:0/3370400268 learned_addr learned my addr 172.21.15.155:0/3370400268 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:37.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 1 -- 172.21.15.155:0/3370400268 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150893097 0 0) 0x7fb260145eb0 con 0x7fb26013f650 2024-03-29T11:44:37.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:37.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:37.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 1 -- 172.21.15.155:0/3370400268 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb250003520 con 0x7fb26013f650 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 1 -- 172.21.15.155:0/3370400268 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681583769 0 0) 0x7fb260145bd0 con 0x7fb2600a9d70 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.789+0000 7fb25f7fe640 1 -- 172.21.15.155:0/3370400268 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb260145eb0 con 0x7fb2600a9d70 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 1 -- 172.21.15.155:0/3370400268 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1751816718 0 0) 0x7fb260145eb0 con 0x7fb2600a9d70 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 1 -- 172.21.15.155:0/3370400268 >> v1:172.21.15.155:6790/0 conn(0x7fb26013f650 legacy=0x7fb2600b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 1 -- 172.21.15.155:0/3370400268 >> v1:172.21.15.155:6789/0 conn(0x7fb2601457f0 legacy=0x7fb26013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:44:37.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 1 -- 172.21.15.155:0/3370400268 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb260146340 con 0x7fb2600a9d70 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 1 -- 172.21.15.155:0/3370400268 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb2580018a0 con 0x7fb2600a9d70 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb25f7fe640 10 monclient: dump: 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:37.800 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:37.801 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:37.801 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:37.801 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:37.801 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:37.801 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:37.801 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:37.801 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:37.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb266a81640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:44:37.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.790+0000 7fb266a81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:37.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.792+0000 7fb25f7fe640 1 -- 172.21.15.155:0/3370400268 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb258001a00 con 0x7fb2600a9d70 2024-03-29T11:44:37.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.792+0000 7fb25f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:37.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.792+0000 7fb25f7fe640 1 -- 172.21.15.155:0/3370400268 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb258001e10 con 0x7fb2600a9d70 2024-03-29T11:44:37.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.792+0000 7fb25f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:37.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.792+0000 7fb25f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:37.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.792+0000 7fb25f7fe640 10 monclient: dump: 2024-03-29T11:44:37.802 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:37.802 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:37.802 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:37.802 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:37.802 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:37.803 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:37.803 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:37.803 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:37.803 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:37.803 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:37.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.792+0000 7fb266a81640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:37.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.792+0000 7fb266a81640 4 set_mon_vals no callback set 2024-03-29T11:44:37.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.793+0000 7fb266a81640 10 monclient: shutdown 2024-03-29T11:44:37.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.793+0000 7fb266a81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:37.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.793+0000 7fb266a81640 1 -- 172.21.15.155:0/3370400268 >> v1:172.21.15.183:6789/0 conn(0x7fb2600a9d70 legacy=0x7fb26014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:37.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.795+0000 7fb266a81640 1 -- 172.21.15.155:0/3370400268 shutdown_connections 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.795+0000 7fb266a81640 1 -- 172.21.15.155:0/3370400268 >> 172.21.15.155:0/3370400268 conn(0x7fb260066a00 msgr2=0x7fb260066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.795+0000 7fb266a81640 1 -- 172.21.15.155:0/3370400268 shutdown_connections 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.795+0000 7fb266a81640 1 -- 172.21.15.155:0/3370400268 wait complete. 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.796+0000 7fb266a81640 10 monclient: build_initial_monmap 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.796+0000 7fb266a81640 10 monclient: monmap: 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:44:37.806 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.796+0000 7fb266a81640 1 librados: starting msgr at 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.796+0000 7fb266a81640 1 librados: starting objecter 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.796+0000 7fb266a81640 1 Processor -- start 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.796+0000 7fb266a81640 1 -- start start 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.796+0000 7fb266a81640 1 librados: setting wanted keys 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.796+0000 7fb266a81640 1 librados: calling monclient init 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.796+0000 7fb266a81640 10 monclient: init 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.797+0000 7fb266a81640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.797+0000 7fb266a81640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.798+0000 7fb266a81640 10 monclient(hunting): picked mon.noname-b con 0x7fb2600a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:37.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.798+0000 7fb266a81640 10 monclient(hunting): picked mon.noname-c con 0x7fb26013f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.801+0000 7fb25ffff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fb26013f650 0x7fb2601db5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50430/0 (socket says 172.21.15.155:50430) 2024-03-29T11:44:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.801+0000 7fb26481d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb2600a9d70 0x7fb2600707a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:49227/0 (socket says 172.21.15.155:49227) 2024-03-29T11:44:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb266a81640 10 monclient(hunting): picked mon.noname-a con 0x7fb2601457f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb266a81640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26013c8d0 con 0x7fb2601457f0 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb266a81640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2600719c0 con 0x7fb2600a9d70 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb266a81640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2600714b0 con 0x7fb26013f650 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb266a81640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb266a81640 10 monclient(hunting): authenticate will time out at 1459.929121s 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb25ffff640 1 -- 172.21.15.155:0/2349962376 learned_addr learned my addr 172.21.15.155:0/2349962376 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb25d7fa640 10 client.?.objecter ms_handle_connect 0x7fb26013f650 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb25d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403165490 0 0) 0x7fb2600714b0 con 0x7fb26013f650 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb25d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb25d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb25d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb25d7fa640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:44:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.802+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb244004eb0 con 0x7fb26013f650 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.803+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1906835181 0 0) 0x7fb244004eb0 con 0x7fb26013f650 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.803+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2600714b0 con 0x7fb26013f650 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.803+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb24c0019d0 con 0x7fb26013f650 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.803+0000 7fb25d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.803+0000 7fb25d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.803+0000 7fb25d7fa640 10 monclient(hunting): dump: 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:37.813 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:37.814 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:37.814 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:37.814 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:37.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.803+0000 7fb25d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:37.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.803+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 610018260 0 0) 0x7fb2600714b0 con 0x7fb26013f650 2024-03-29T11:44:37.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.803+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 >> v1:172.21.15.183:6789/0 conn(0x7fb2600a9d70 legacy=0x7fb2600707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:37.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.803+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 >> v1:172.21.15.155:6789/0 conn(0x7fb2601457f0 legacy=0x7fb2601e5b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:37.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.804+0000 7fb26481d640 1 --1- 172.21.15.155:0/2349962376 >> v1:172.21.15.183:6789/0 conn(0x7fb2600a9d70 0x7fb2600707a0 :-1 s=CLOSED pgs=0 cs=0 l=1).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-29T11:44:37.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb25d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:44:37.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb25d7fa640 1 monclient: found mon.c 2024-03-29T11:44:37.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb25d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:37.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb25d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:37.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb260071640 con 0x7fb26013f650 2024-03-29T11:44:37.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb25d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:44:37.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb25d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:44:37.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb25d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:37.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb266a81640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:44:37.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb266a81640 10 monclient: _renew_subs 2024-03-29T11:44:37.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb24c001c30 con 0x7fb26013f650 2024-03-29T11:44:37.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb266a81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:37.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.806+0000 7fb266a81640 1 -- 172.21.15.155:0/2349962376 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2601dbce0 con 0x7fb26013f650 2024-03-29T11:44:37.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.807+0000 7fb25d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:37.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.807+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fb24c004c70 con 0x7fb26013f650 2024-03-29T11:44:37.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.807+0000 7fb25d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:37.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.807+0000 7fb25d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:37.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.807+0000 7fb25d7fa640 10 monclient: dump: 2024-03-29T11:44:37.819 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.807+0000 7fb25effd640 4 set_mon_vals no callback set 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.808+0000 7fb266a81640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.808+0000 7fb266a81640 10 monclient: _renew_subs 2024-03-29T11:44:37.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.808+0000 7fb266a81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:37.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.808+0000 7fb266a81640 1 -- 172.21.15.155:0/2349962376 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2601e7260 con 0x7fb26013f650 2024-03-29T11:44:37.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.808+0000 7fb266a81640 1 librados: init done 2024-03-29T11:44:37.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.809+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fb24c004eb0 con 0x7fb26013f650 2024-03-29T11:44:37.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.810+0000 7fb24affd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:37.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.810+0000 7fb24affd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:37.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.810+0000 7fb24affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:37.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.810+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 339942357 0 0) 0x7fb24c085eb0 con 0x7fb26013f650 2024-03-29T11:44:37.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.810+0000 7fb25d7fa640 10 client.4649.objecter ms_dispatch 0x7fb260000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:44:37.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.810+0000 7fb25d7fa640 3 client.4649.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:44:37.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.810+0000 7fb25d7fa640 3 client.4649.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:44:37.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.810+0000 7fb25d7fa640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-29T11:44:37.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.810+0000 7fb24affd640 1 -- 172.21.15.155:0/2349962376 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2601db2f0 con 0x7fb26013f650 2024-03-29T11:44:37.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.810+0000 7fb26481d640 10 monclient: get_auth_request con 0x7fb24406db20 auth_method 0 2024-03-29T11:44:37.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.811+0000 7fb25d7fa640 10 client.4649.objecter ms_handle_connect 0x7fb24406db20 2024-03-29T11:44:37.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.818+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2601db2f0 con 0x7fb26013f650 2024-03-29T11:44:37.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.818+0000 7fb25d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:37.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:37.818+0000 7fb25d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:38.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.134+0000 7fb24affd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:38.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.134+0000 7fb24affd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:38.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.134+0000 7fb24affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:38.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.134+0000 7fb24affd640 1 -- 172.21.15.155:0/2349962376 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2601db2f0 con 0x7fb26013f650 2024-03-29T11:44:38.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.137+0000 7fb25d7fa640 1 -- 172.21.15.155:0/2349962376 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1459448867) 0x7fb2601db2f0 con 0x7fb26013f650 2024-03-29T11:44:38.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.137+0000 7fb25d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:38.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.137+0000 7fb25d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:38.143 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:44:38.143 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:44:36.686112+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:36.686112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:38.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.140+0000 7fb24affd640 10 librados: watch_flush enter 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.140+0000 7fb24affd640 10 librados: watch_flush exit 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.141+0000 7fb24affd640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.141+0000 7fb24affd640 10 client.4649.objecter successfully canceled tick 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.141+0000 7fb24affd640 1 -- 172.21.15.155:0/2349962376 >> v1:172.21.15.155:6816/1760814708 conn(0x7fb24406db20 legacy=0x7fb24406ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.141+0000 7fb24affd640 10 monclient: shutdown 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.141+0000 7fb24affd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.141+0000 7fb24affd640 1 -- 172.21.15.155:0/2349962376 >> v1:172.21.15.155:6790/0 conn(0x7fb26013f650 legacy=0x7fb2601db5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.141+0000 7fb24affd640 1 -- 172.21.15.155:0/2349962376 shutdown_connections 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.141+0000 7fb24affd640 1 -- 172.21.15.155:0/2349962376 >> 172.21.15.155:0/2349962376 conn(0x7fb260066a00 msgr2=0x7fb260066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.141+0000 7fb24affd640 1 -- 172.21.15.155:0/2349962376 shutdown_connections 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.141+0000 7fb24affd640 1 -- 172.21.15.155:0/2349962376 wait complete. 2024-03-29T11:44:38.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:38.141+0000 7fb24affd640 1 librados: shutdown 2024-03-29T11:44:38.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:44.972 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:45.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.113+0000 7f63c817a640 10 monclient: get_monmap_and_config 2024-03-29T11:44:45.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.113+0000 7f63c817a640 10 monclient: build_initial_monmap 2024-03-29T11:44:45.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.113+0000 7f63c817a640 10 monclient: monmap: 2024-03-29T11:44:45.118 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:45.118 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:45.118 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:44:45.115128+0000 2024-03-29T11:44:45.118 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:44:45.115128+0000 2024-03-29T11:44:45.118 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:45.119 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:45.119 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:45.119 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:44:45.119 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:44:45.119 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:45.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.117+0000 7f63c817a640 1 Processor -- start 2024-03-29T11:44:45.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.117+0000 7f63c817a640 1 -- start start 2024-03-29T11:44:45.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.117+0000 7f63c817a640 10 monclient: init 2024-03-29T11:44:45.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c817a640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:45.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c817a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:45.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c817a640 10 monclient(hunting): picked mon.noname-c con 0x7f63c01457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:45.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c817a640 10 monclient(hunting): picked mon.noname-b con 0x7f63c013f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:45.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c817a640 10 monclient(hunting): picked mon.noname-a con 0x7f63c00a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:45.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c817a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c00aaef0 con 0x7f63c00a9d70 2024-03-29T11:44:45.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c817a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c0145bd0 con 0x7f63c01457f0 2024-03-29T11:44:45.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c817a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c0145eb0 con 0x7f63c013f650 2024-03-29T11:44:45.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c817a640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:45.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c817a640 10 monclient(hunting): authenticate will time out at 1467.245557s 2024-03-29T11:44:45.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c5715640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f63c013f650 0x7f63c00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:38904/0 (socket says 172.21.15.155:38904) 2024-03-29T11:44:45.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.118+0000 7f63c5715640 1 -- 172.21.15.155:0/3375405775 learned_addr learned my addr 172.21.15.155:0/3375405775 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:45.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.121+0000 7f63c4f14640 1 -- 172.21.15.155:0/3375405775 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394729816 0 0) 0x7f63c0145eb0 con 0x7f63c013f650 2024-03-29T11:44:45.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.121+0000 7f63c4f14640 10 monclient(hunting): _negotiate 2024-03-29T11:44:45.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.121+0000 7f63c4f14640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:45.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.121+0000 7f63c4f14640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:45.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.121+0000 7f63c4f14640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:44:45.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.121+0000 7f63c4f14640 1 -- 172.21.15.155:0/3375405775 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac003520 con 0x7f63c013f650 2024-03-29T11:44:45.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 1 -- 172.21.15.155:0/3375405775 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3500431926 0 0) 0x7f63ac003520 con 0x7f63c013f650 2024-03-29T11:44:45.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 1 -- 172.21.15.155:0/3375405775 >> v1:172.21.15.183:6789/0 conn(0x7f63c01457f0 legacy=0x7f63c013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:45.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 1 -- 172.21.15.155:0/3375405775 >> v1:172.21.15.155:6789/0 conn(0x7f63c00a9d70 legacy=0x7f63c014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:45.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 10 monclient: _finish_hunting 0 2024-03-29T11:44:45.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 1 monclient: found mon.noname-b 2024-03-29T11:44:45.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:45.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:44:45.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 1 -- 172.21.15.155:0/3375405775 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0146340 con 0x7f63c013f650 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 10 monclient: _finish_auth 0 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 10 monclient: _check_auth_tickets 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 1 -- 172.21.15.155:0/3375405775 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f63bc001670 con 0x7f63c013f650 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c4f14640 10 monclient: dump: 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:45.129 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c817a640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.122+0000 7f63c817a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.123+0000 7f63c4f14640 1 -- 172.21.15.155:0/3375405775 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63bc0018d0 con 0x7f63c013f650 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.123+0000 7f63c4f14640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.123+0000 7f63c4f14640 1 -- 172.21.15.155:0/3375405775 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f63bc001d20 con 0x7f63c013f650 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.123+0000 7f63c4f14640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.123+0000 7f63c4f14640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.123+0000 7f63c4f14640 10 monclient: dump: 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:45.130 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.123+0000 7f63c817a640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.123+0000 7f63c817a640 4 set_mon_vals no callback set 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.124+0000 7f63c817a640 10 monclient: shutdown 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.124+0000 7f63c817a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.124+0000 7f63c817a640 1 -- 172.21.15.155:0/3375405775 >> v1:172.21.15.155:6790/0 conn(0x7f63c013f650 legacy=0x7f63c00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.124+0000 7f63c817a640 1 -- 172.21.15.155:0/3375405775 shutdown_connections 2024-03-29T11:44:45.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.124+0000 7f63c817a640 1 -- 172.21.15.155:0/3375405775 >> 172.21.15.155:0/3375405775 conn(0x7f63c0066a00 msgr2=0x7f63c0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:45.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.126+0000 7f63c817a640 1 -- 172.21.15.155:0/3375405775 shutdown_connections 2024-03-29T11:44:45.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.126+0000 7f63c817a640 1 -- 172.21.15.155:0/3375405775 wait complete. 2024-03-29T11:44:45.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.127+0000 7f63c817a640 10 monclient: build_initial_monmap 2024-03-29T11:44:45.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.127+0000 7f63c817a640 10 monclient: monmap: 2024-03-29T11:44:45.132 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:45.132 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:45.132 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:44:45.132 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:44:45.133 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:45.133 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:45.133 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:45.133 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:44:45.133 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:44:45.133 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:45.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.128+0000 7f63c817a640 1 librados: starting msgr at 2024-03-29T11:44:45.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.128+0000 7f63c817a640 1 librados: starting objecter 2024-03-29T11:44:45.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.128+0000 7f63c817a640 1 Processor -- start 2024-03-29T11:44:45.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.128+0000 7f63c817a640 1 -- start start 2024-03-29T11:44:45.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.128+0000 7f63c817a640 1 librados: setting wanted keys 2024-03-29T11:44:45.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.128+0000 7f63c817a640 1 librados: calling monclient init 2024-03-29T11:44:45.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.128+0000 7f63c817a640 10 monclient: init 2024-03-29T11:44:45.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.128+0000 7f63c817a640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:45.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.128+0000 7f63c817a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:45.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.129+0000 7f63c817a640 10 monclient(hunting): picked mon.noname-b con 0x7f63c00a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:45.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.129+0000 7f63c817a640 10 monclient(hunting): picked mon.noname-a con 0x7f63c01457f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:45.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.129+0000 7f63c817a640 10 monclient(hunting): picked mon.noname-c con 0x7f63c013b9d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:45.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.129+0000 7f63c817a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c0135b20 con 0x7f63c01457f0 2024-03-29T11:44:45.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.129+0000 7f63c817a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c013b140 con 0x7f63c00a9d70 2024-03-29T11:44:45.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.129+0000 7f63c817a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c0138dc0 con 0x7f63c013b9d0 2024-03-29T11:44:45.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.129+0000 7f63c817a640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:45.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.129+0000 7f63c817a640 10 monclient(hunting): authenticate will time out at 1467.255949s 2024-03-29T11:44:45.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.132+0000 7f63c6717640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f63c013b9d0 0x7f63c01e5f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:44234/0 (socket says 172.21.15.155:44234) 2024-03-29T11:44:45.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.132+0000 7f63c6717640 1 -- 172.21.15.155:0/4012817104 learned_addr learned my addr 172.21.15.155:0/4012817104 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:45.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.132+0000 7f63b67fc640 10 client.?.objecter ms_handle_connect 0x7f63c013b9d0 2024-03-29T11:44:45.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.132+0000 7f63b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:45.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.132+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826570125 0 0) 0x7f63c0138dc0 con 0x7f63c013b9d0 2024-03-29T11:44:45.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.132+0000 7f63b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:45.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.132+0000 7f63b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:45.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.132+0000 7f63b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:45.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.132+0000 7f63b67fc640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:44:45.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.132+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a4004eb0 con 0x7f63c013b9d0 2024-03-29T11:44:45.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72229282 0 0) 0x7f63a4004eb0 con 0x7f63c013b9d0 2024-03-29T11:44:45.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63c0138dc0 con 0x7f63c013b9d0 2024-03-29T11:44:45.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 10 client.?.objecter ms_handle_connect 0x7f63c01457f0 2024-03-29T11:44:45.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:45.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f63b8001c60 con 0x7f63c013b9d0 2024-03-29T11:44:45.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:45.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:45.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 10 monclient(hunting): dump: 2024-03-29T11:44:45.147 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:45.147 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:45.147 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:45.147 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:45.147 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:45.147 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:45.147 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:45.147 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:45.148 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:45.148 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:45.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:45.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365869073 0 0) 0x7f63c0135b20 con 0x7f63c01457f0 2024-03-29T11:44:45.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:45.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:45.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:45.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a4004eb0 con 0x7f63c01457f0 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 919448691 0 0) 0x7f63c0138dc0 con 0x7f63c013b9d0 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.134+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 >> v1:172.21.15.183:6789/0 conn(0x7f63c00a9d70 legacy=0x7f63c0139b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 >> v1:172.21.15.155:6789/0 conn(0x7f63c01457f0 legacy=0x7f63c01405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 1 monclient: found mon.c 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0071fc0 con 0x7f63c013b9d0 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63c817a640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:44:45.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63c817a640 10 monclient: _renew_subs 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63c817a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63c817a640 1 -- 172.21.15.155:0/4012817104 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c013a270 con 0x7f63c013b9d0 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63c817a640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63c817a640 10 monclient: _renew_subs 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63c817a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63c817a640 1 -- 172.21.15.155:0/4012817104 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c01e7860 con 0x7f63c013b9d0 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63c817a640 1 librados: init done 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63b80020b0 con 0x7f63c013b9d0 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f63b8003c20 con 0x7f63c013b9d0 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.135+0000 7f63b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:45.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.136+0000 7f63b67fc640 10 monclient: dump: 2024-03-29T11:44:45.151 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:45.151 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:45.151 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:45.151 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:45.151 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:45.151 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:45.151 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:45.151 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.136+0000 7f63b77fe640 4 set_mon_vals no callback set 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.137+0000 7f63c817a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.137+0000 7f63c817a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.137+0000 7f63c817a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.137+0000 7f63c817a640 1 -- 172.21.15.155:0/4012817104 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a8005080 con 0x7f63c013b9d0 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.138+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f63b8003e80 con 0x7f63c013b9d0 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.139+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 339942357 0 0) 0x7f63b8084e20 con 0x7f63c013b9d0 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.139+0000 7f63b67fc640 10 client.4655.objecter ms_dispatch 0x7f63c0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.139+0000 7f63b67fc640 3 client.4655.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:44:45.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.139+0000 7f63b67fc640 3 client.4655.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:44:45.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.139+0000 7f63c5f16640 10 monclient: get_auth_request con 0x7f63a406dce0 auth_method 0 2024-03-29T11:44:45.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.139+0000 7f63b67fc640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-29T11:44:45.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.139+0000 7f63b67fc640 10 client.4655.objecter ms_handle_connect 0x7f63a406dce0 2024-03-29T11:44:45.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.148+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63b8050700 con 0x7f63c013b9d0 2024-03-29T11:44:45.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.148+0000 7f63b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:45.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.148+0000 7f63b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:45.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.471+0000 7f63c817a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:45.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.471+0000 7f63c817a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:45.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.471+0000 7f63c817a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:45.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.471+0000 7f63c817a640 1 -- 172.21.15.155:0/4012817104 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63a8005330 con 0x7f63c013b9d0 2024-03-29T11:44:45.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.472+0000 7f63b67fc640 1 -- 172.21.15.155:0/4012817104 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1459448867) 0x7f63b8052630 con 0x7f63c013b9d0 2024-03-29T11:44:45.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.472+0000 7f63b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:45.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.472+0000 7f63b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:45.477 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:44:45.478 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:44:36.686112+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:36.686112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:45.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.476+0000 7f63c817a640 10 librados: watch_flush enter 2024-03-29T11:44:45.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.476+0000 7f63c817a640 10 librados: watch_flush exit 2024-03-29T11:44:45.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.476+0000 7f63c817a640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-29T11:44:45.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.476+0000 7f63c817a640 10 client.4655.objecter successfully canceled tick 2024-03-29T11:44:45.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.476+0000 7f63c817a640 1 -- 172.21.15.155:0/4012817104 >> v1:172.21.15.155:6816/1760814708 conn(0x7f63a406dce0 legacy=0x7f63a40701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:45.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.476+0000 7f63c817a640 10 monclient: shutdown 2024-03-29T11:44:45.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.476+0000 7f63c817a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:45.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.476+0000 7f63c817a640 1 -- 172.21.15.155:0/4012817104 >> v1:172.21.15.155:6790/0 conn(0x7f63c013b9d0 legacy=0x7f63c01e5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:45.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.477+0000 7f63c817a640 1 -- 172.21.15.155:0/4012817104 shutdown_connections 2024-03-29T11:44:45.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.477+0000 7f63c817a640 1 -- 172.21.15.155:0/4012817104 >> 172.21.15.155:0/4012817104 conn(0x7f63c0066a00 msgr2=0x7f63c0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:45.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.479+0000 7f63c817a640 1 -- 172.21.15.155:0/4012817104 shutdown_connections 2024-03-29T11:44:45.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.479+0000 7f63c817a640 1 -- 172.21.15.155:0/4012817104 wait complete. 2024-03-29T11:44:45.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:45.479+0000 7f63c817a640 1 librados: shutdown 2024-03-29T11:44:45.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:52.306 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:52.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.455+0000 7f30d2fef640 10 monclient: get_monmap_and_config 2024-03-29T11:44:52.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.455+0000 7f30d2fef640 10 monclient: build_initial_monmap 2024-03-29T11:44:52.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.455+0000 7f30d2fef640 10 monclient: monmap: 2024-03-29T11:44:52.460 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:52.460 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:52.460 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:44:52.456507+0000 2024-03-29T11:44:52.460 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:44:52.456507+0000 2024-03-29T11:44:52.460 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:52.460 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:52.460 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:52.461 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:44:52.461 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:44:52.461 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:52.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.456+0000 7f30d2fef640 1 Processor -- start 2024-03-29T11:44:52.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.456+0000 7f30d2fef640 1 -- start start 2024-03-29T11:44:52.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.456+0000 7f30d2fef640 10 monclient: init 2024-03-29T11:44:52.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.457+0000 7f30d2fef640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:52.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.457+0000 7f30d2fef640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:52.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.458+0000 7f30d2fef640 10 monclient(hunting): picked mon.noname-b con 0x7f30cc0a9ac0 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:52.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.458+0000 7f30d2fef640 10 monclient(hunting): picked mon.noname-a con 0x7f30cc13ee90 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:52.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.458+0000 7f30d2fef640 10 monclient(hunting): picked mon.noname-c con 0x7f30cc0b4100 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:52.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.458+0000 7f30d2fef640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc0af6f0 con 0x7f30cc13ee90 2024-03-29T11:44:52.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.458+0000 7f30d2fef640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc0aac80 con 0x7f30cc0b4100 2024-03-29T11:44:52.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.458+0000 7f30d2fef640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc0aaf60 con 0x7f30cc0a9ac0 2024-03-29T11:44:52.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.458+0000 7f30d2fef640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:52.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.458+0000 7f30d2fef640 10 monclient(hunting): authenticate will time out at 1474.585013s 2024-03-29T11:44:52.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.458+0000 7f30d0d8b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f30cc0a9ac0 0x7f30cc0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:40954/0 (socket says 172.21.15.155:40954) 2024-03-29T11:44:52.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.458+0000 7f30d0d8b640 1 -- 172.21.15.155:0/4264879569 learned_addr learned my addr 172.21.15.155:0/4264879569 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:52.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 1 -- 172.21.15.155:0/4264879569 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058082642 0 0) 0x7f30cc0af6f0 con 0x7f30cc13ee90 2024-03-29T11:44:52.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:52.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:52.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:52.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 1 -- 172.21.15.155:0/4264879569 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30bc003520 con 0x7f30cc13ee90 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 1 -- 172.21.15.155:0/4264879569 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261623511 0 0) 0x7f30cc0aaf60 con 0x7f30cc0a9ac0 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.462+0000 7f30cb7fe640 1 -- 172.21.15.155:0/4264879569 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30cc0af6f0 con 0x7f30cc0a9ac0 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 1 -- 172.21.15.155:0/4264879569 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2201564445 0 0) 0x7f30bc003520 con 0x7f30cc13ee90 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 1 -- 172.21.15.155:0/4264879569 >> v1:172.21.15.155:6790/0 conn(0x7f30cc0a9ac0 legacy=0x7f30cc0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 1 -- 172.21.15.155:0/4264879569 >> v1:172.21.15.183:6789/0 conn(0x7f30cc0b4100 legacy=0x7f30cc14a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 1 monclient: found mon.noname-a 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:52.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 1 -- 172.21.15.155:0/4264879569 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc13e970 con 0x7f30cc13ee90 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 1 -- 172.21.15.155:0/4264879569 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f30b8001690 con 0x7f30cc13ee90 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 10 monclient: dump: 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:52.470 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 1 -- 172.21.15.155:0/4264879569 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30b80017f0 con 0x7f30cc13ee90 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30d2fef640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30d2fef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 1 -- 172.21.15.155:0/4264879569 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f30b8001c40 con 0x7f30cc13ee90 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30d2fef640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30d2fef640 4 set_mon_vals no callback set 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:52.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:52.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.463+0000 7f30cb7fe640 10 monclient: dump: 2024-03-29T11:44:52.472 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:52.472 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:52.472 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:52.472 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:52.472 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:52.472 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:52.472 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:52.472 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:52.472 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:52.472 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:52.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.464+0000 7f30d2fef640 10 monclient: shutdown 2024-03-29T11:44:52.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.464+0000 7f30d2fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:52.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.464+0000 7f30d2fef640 1 -- 172.21.15.155:0/4264879569 >> v1:172.21.15.155:6789/0 conn(0x7f30cc13ee90 legacy=0x7f30cc13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:52.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.464+0000 7f30d2fef640 1 -- 172.21.15.155:0/4264879569 shutdown_connections 2024-03-29T11:44:52.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.464+0000 7f30d2fef640 1 -- 172.21.15.155:0/4264879569 >> 172.21.15.155:0/4264879569 conn(0x7f30cc0b35e0 msgr2=0x7f30cc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.465+0000 7f30d2fef640 1 -- 172.21.15.155:0/4264879569 shutdown_connections 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.465+0000 7f30d2fef640 1 -- 172.21.15.155:0/4264879569 wait complete. 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.465+0000 7f30d2fef640 10 monclient: build_initial_monmap 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.466+0000 7f30d2fef640 10 monclient: monmap: 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:44:52.475 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:52.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.466+0000 7f30d2fef640 1 librados: starting msgr at 2024-03-29T11:44:52.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.466+0000 7f30d2fef640 1 librados: starting objecter 2024-03-29T11:44:52.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.466+0000 7f30d2fef640 1 Processor -- start 2024-03-29T11:44:52.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.466+0000 7f30d2fef640 1 -- start start 2024-03-29T11:44:52.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.466+0000 7f30d2fef640 1 librados: setting wanted keys 2024-03-29T11:44:52.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.466+0000 7f30d2fef640 1 librados: calling monclient init 2024-03-29T11:44:52.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.466+0000 7f30d2fef640 10 monclient: init 2024-03-29T11:44:52.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.467+0000 7f30d2fef640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:52.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.467+0000 7f30d2fef640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:52.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.467+0000 7f30d2fef640 10 monclient(hunting): picked mon.noname-b con 0x7f30cc0a9ac0 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.467+0000 7f30d2fef640 10 monclient(hunting): picked mon.noname-c con 0x7f30cc0b4100 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.467+0000 7f30cbfff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f30cc0b4100 0x7f30cc1411f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:41800/0 (socket says 172.21.15.155:41800) 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.467+0000 7f30cbfff640 1 -- 172.21.15.155:0/1123519673 learned_addr learned my addr 172.21.15.155:0/1123519673 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.468+0000 7f30d2fef640 10 monclient(hunting): picked mon.noname-a con 0x7f30cc141900 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.468+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc1d6560 con 0x7f30cc141900 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.468+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc1d6840 con 0x7f30cc0a9ac0 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.468+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc1d73e0 con 0x7f30cc0b4100 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.468+0000 7f30d2fef640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.468+0000 7f30d2fef640 10 monclient(hunting): authenticate will time out at 1474.595018s 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.471+0000 7f30c97fa640 10 client.?.objecter ms_handle_connect 0x7f30cc0b4100 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.471+0000 7f30c97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:52.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.471+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791072670 0 0) 0x7f30cc1d73e0 con 0x7f30cc0b4100 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.471+0000 7f30c97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.471+0000 7f30c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.471+0000 7f30c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.471+0000 7f30c97fa640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.471+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b0004eb0 con 0x7f30cc0b4100 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.472+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2662478345 0 0) 0x7f30b0004eb0 con 0x7f30cc0b4100 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.472+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30cc1d73e0 con 0x7f30cc0b4100 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.472+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f30b8001bc0 con 0x7f30cc0b4100 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.472+0000 7f30c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.472+0000 7f30c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.472+0000 7f30c97fa640 10 monclient(hunting): dump: 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:52.484 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.472+0000 7f30c97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.473+0000 7f30c97fa640 10 client.?.objecter ms_handle_connect 0x7f30cc141900 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.473+0000 7f30c97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.473+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084597698 0 0) 0x7f30cc1d6560 con 0x7f30cc141900 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.473+0000 7f30c97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.473+0000 7f30c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.473+0000 7f30c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.473+0000 7f30c97fa640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:44:52.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.473+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b0004eb0 con 0x7f30cc141900 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.473+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3275395954 0 0) 0x7f30cc1d73e0 con 0x7f30cc0b4100 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.473+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 >> v1:172.21.15.183:6789/0 conn(0x7f30cc0a9ac0 legacy=0x7f30cc1d6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.473+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 >> v1:172.21.15.155:6789/0 conn(0x7f30cc141900 legacy=0x7f30cc1ddb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30c97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30c97fa640 1 monclient: found mon.c 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30c97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc1405e0 con 0x7f30cc0b4100 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30c97fa640 10 monclient: _finish_auth 0 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30c97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30d2fef640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30d2fef640 10 monclient: _renew_subs 2024-03-29T11:44:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30d2fef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:52.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30cc13ccc0 con 0x7f30cc0b4100 2024-03-29T11:44:52.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30d2fef640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:52.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30d2fef640 10 monclient: _renew_subs 2024-03-29T11:44:52.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.474+0000 7f30d2fef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:52.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.475+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30cc1df290 con 0x7f30cc0b4100 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.475+0000 7f30d2fef640 1 librados: init done 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.475+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30b8003940 con 0x7f30cc0b4100 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.475+0000 7f30c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.475+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f30b8003d50 con 0x7f30cc0b4100 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.475+0000 7f30c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.475+0000 7f30c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.475+0000 7f30c97fa640 10 monclient: dump: 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:52.490 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:52.491 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:52.491 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.475+0000 7f30caffd640 4 set_mon_vals no callback set 2024-03-29T11:44:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.476+0000 7f30d2fef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.476+0000 7f30d2fef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.477+0000 7f30d2fef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.477+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ac005080 con 0x7f30cc0b4100 2024-03-29T11:44:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.477+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f30b8003eb0 con 0x7f30cc0b4100 2024-03-29T11:44:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.478+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 339942357 0 0) 0x7f30b8084bd0 con 0x7f30cc0b4100 2024-03-29T11:44:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.478+0000 7f30c97fa640 10 client.4661.objecter ms_dispatch 0x7f30cc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:44:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.478+0000 7f30c97fa640 3 client.4661.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:44:52.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.478+0000 7f30d0d8b640 10 monclient: get_auth_request con 0x7f30b006dd00 auth_method 0 2024-03-29T11:44:52.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.479+0000 7f30c97fa640 3 client.4661.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:44:52.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.479+0000 7f30c97fa640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T11:44:52.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.479+0000 7f30c97fa640 10 client.4661.objecter ms_handle_connect 0x7f30b006dd00 2024-03-29T11:44:52.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.485+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30b80504b0 con 0x7f30cc0b4100 2024-03-29T11:44:52.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.485+0000 7f30c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:52.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.485+0000 7f30c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:52.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.822+0000 7f30d2fef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:52.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.822+0000 7f30d2fef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:52.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.822+0000 7f30d2fef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:52.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.822+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30ac005330 con 0x7f30cc0b4100 2024-03-29T11:44:52.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.824+0000 7f30c97fa640 1 -- 172.21.15.155:0/1123519673 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1459448867) 0x7f30b80523e0 con 0x7f30cc0b4100 2024-03-29T11:44:52.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.824+0000 7f30c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:52.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.824+0000 7f30c97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:52.829 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:44:52.829 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:44:36.686112+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:36.686112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:52.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.828+0000 7f30d2fef640 10 librados: watch_flush enter 2024-03-29T11:44:52.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.828+0000 7f30d2fef640 10 librados: watch_flush exit 2024-03-29T11:44:52.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.828+0000 7f30d2fef640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T11:44:52.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.828+0000 7f30d2fef640 10 client.4661.objecter successfully canceled tick 2024-03-29T11:44:52.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.828+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 >> v1:172.21.15.155:6816/1760814708 conn(0x7f30b006dd00 legacy=0x7f30b00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:52.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.829+0000 7f30d2fef640 10 monclient: shutdown 2024-03-29T11:44:52.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.829+0000 7f30d2fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:52.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.829+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 >> v1:172.21.15.155:6790/0 conn(0x7f30cc0b4100 legacy=0x7f30cc1411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:52.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.830+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 shutdown_connections 2024-03-29T11:44:52.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.830+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 >> 172.21.15.155:0/1123519673 conn(0x7f30cc0b35e0 msgr2=0x7f30cc0b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:52.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.830+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 shutdown_connections 2024-03-29T11:44:52.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.830+0000 7f30d2fef640 1 -- 172.21.15.155:0/1123519673 wait complete. 2024-03-29T11:44:52.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:52.830+0000 7f30d2fef640 1 librados: shutdown 2024-03-29T11:44:52.852 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:59.658 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.785+0000 7fada8862640 10 monclient: get_monmap_and_config 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.785+0000 7fada8862640 10 monclient: build_initial_monmap 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.785+0000 7fada8862640 10 monclient: monmap: 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:44:59.785869+0000 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:44:59.785869+0000 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:44:59.789 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:44:59.790 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:59.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.786+0000 7fada8862640 1 Processor -- start 2024-03-29T11:44:59.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.786+0000 7fada8862640 1 -- start start 2024-03-29T11:44:59.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.786+0000 7fada8862640 10 monclient: init 2024-03-29T11:44:59.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada8862640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada8862640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada8862640 10 monclient(hunting): picked mon.noname-b con 0x7fada40b2800 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada8862640 10 monclient(hunting): picked mon.noname-a con 0x7fada40a9c00 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada8862640 10 monclient(hunting): picked mon.noname-c con 0x7fada40b38c0 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada8862640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada4141eb0 con 0x7fada40a9c00 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada8862640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada40b22d0 con 0x7fada40b38c0 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada8862640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada4142d30 con 0x7fada40b2800 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada8862640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada8862640 10 monclient(hunting): authenticate will time out at 1481.913272s 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada2d9d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fada40b2800 0x7fada4142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39960/0 (socket says 172.21.15.155:39960) 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada2d9d640 1 -- 172.21.15.155:0/1491432149 learned_addr learned my addr 172.21.15.155:0/1491432149 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada259c640 1 -- 172.21.15.155:0/1491432149 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542434905 0 0) 0x7fada4142d30 con 0x7fada40b2800 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada259c640 10 monclient(hunting): _negotiate 2024-03-29T11:44:59.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada259c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada259c640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.787+0000 7fada259c640 1 -- 172.21.15.155:0/1491432149 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad90003520 con 0x7fada40b2800 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.788+0000 7fada259c640 1 -- 172.21.15.155:0/1491432149 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656606281 0 0) 0x7fada40b22d0 con 0x7fada40b38c0 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.788+0000 7fada259c640 10 monclient(hunting): _negotiate 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.788+0000 7fada259c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.788+0000 7fada259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.788+0000 7fada259c640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.788+0000 7fada259c640 1 -- 172.21.15.155:0/1491432149 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada4142d30 con 0x7fada40b38c0 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.788+0000 7fada259c640 1 -- 172.21.15.155:0/1491432149 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2744066176 0 0) 0x7fad90003520 con 0x7fada40b2800 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.788+0000 7fada259c640 1 -- 172.21.15.155:0/1491432149 >> v1:172.21.15.183:6789/0 conn(0x7fada40b38c0 legacy=0x7fada414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.788+0000 7fada259c640 1 -- 172.21.15.155:0/1491432149 >> v1:172.21.15.155:6789/0 conn(0x7fada40a9c00 legacy=0x7fada413bb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:59.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 10 monclient: _finish_hunting 0 2024-03-29T11:44:59.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 1 monclient: found mon.noname-b 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 1 -- 172.21.15.155:0/1491432149 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40aaf90 con 0x7fada40b2800 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 10 monclient: _finish_auth 0 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 10 monclient: _check_auth_tickets 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 1 -- 172.21.15.155:0/1491432149 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fad94001690 con 0x7fada40b2800 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada259c640 10 monclient: dump: 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:59.796 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada8862640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.790+0000 7fada8862640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.791+0000 7fada259c640 1 -- 172.21.15.155:0/1491432149 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad94001870 con 0x7fada40b2800 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.791+0000 7fada259c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.791+0000 7fada8862640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.791+0000 7fada8862640 4 set_mon_vals no callback set 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.791+0000 7fada259c640 1 -- 172.21.15.155:0/1491432149 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fad94001c00 con 0x7fada40b2800 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.791+0000 7fada259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.791+0000 7fada259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.791+0000 7fada259c640 10 monclient: dump: 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.792+0000 7fada8862640 10 monclient: shutdown 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.792+0000 7fada8862640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:59.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.792+0000 7fada8862640 1 -- 172.21.15.155:0/1491432149 >> v1:172.21.15.155:6790/0 conn(0x7fada40b2800 legacy=0x7fada4142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:59.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.792+0000 7fada8862640 1 -- 172.21.15.155:0/1491432149 shutdown_connections 2024-03-29T11:44:59.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.792+0000 7fada8862640 1 -- 172.21.15.155:0/1491432149 >> 172.21.15.155:0/1491432149 conn(0x7fada4066a00 msgr2=0x7fada4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:59.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.793+0000 7fada8862640 1 -- 172.21.15.155:0/1491432149 shutdown_connections 2024-03-29T11:44:59.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.793+0000 7fada8862640 1 -- 172.21.15.155:0/1491432149 wait complete. 2024-03-29T11:44:59.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.794+0000 7fada8862640 10 monclient: build_initial_monmap 2024-03-29T11:44:59.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.794+0000 7fada8862640 10 monclient: monmap: 2024-03-29T11:44:59.802 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.794+0000 7fada8862640 1 librados: starting msgr at 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.794+0000 7fada8862640 1 librados: starting objecter 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.795+0000 7fada8862640 1 Processor -- start 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.795+0000 7fada8862640 1 -- start start 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.795+0000 7fada8862640 1 librados: setting wanted keys 2024-03-29T11:44:59.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.795+0000 7fada8862640 1 librados: calling monclient init 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.795+0000 7fada8862640 10 monclient: init 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fada8862640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fada8862640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fada8862640 10 monclient(hunting): picked mon.noname-a con 0x7fada40a9c00 addr v1:172.21.15.155:6789/0 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fada8862640 10 monclient(hunting): picked mon.noname-c con 0x7fada40b2800 addr v1:172.21.15.155:6790/0 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fada8862640 10 monclient(hunting): picked mon.noname-b con 0x7fada40b38c0 addr v1:172.21.15.183:6789/0 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fada8862640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada41d6da0 con 0x7fada40a9c00 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fada8862640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada40afd80 con 0x7fada40b38c0 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fada8862640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada41d7940 con 0x7fada40b2800 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fada8862640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fada8862640 10 monclient(hunting): authenticate will time out at 1481.922626s 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fad9bfff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fada40b2800 0x7fada41d7230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:45944/0 (socket says 172.21.15.155:45944) 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fad9bfff640 1 -- 172.21.15.155:0/4125579339 learned_addr learned my addr 172.21.15.155:0/4125579339 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:44:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.796+0000 7fad9b7fe640 10 client.?.objecter ms_handle_connect 0x7fada40b2800 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991898214 0 0) 0x7fada41d7940 con 0x7fada40b2800 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad84004eb0 con 0x7fada40b2800 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 748687616 0 0) 0x7fad84004eb0 con 0x7fada40b2800 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fada41d7940 con 0x7fada40b2800 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fad8c001930 con 0x7fada40b2800 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 10 monclient(hunting): dump: 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:59.805 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.797+0000 7fad9b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.798+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 503222156 0 0) 0x7fada41d7940 con 0x7fada40b2800 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.798+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 >> v1:172.21.15.183:6789/0 conn(0x7fada40b38c0 legacy=0x7fada41e1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:59.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.798+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 >> v1:172.21.15.155:6789/0 conn(0x7fada40a9c00 legacy=0x7fada40ad990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:59.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fad9b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:59.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fad9b7fe640 1 monclient: found mon.c 2024-03-29T11:44:59.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fad9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:59.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fad9b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:59.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada41d7ad0 con 0x7fada40b2800 2024-03-29T11:44:59.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fad9b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:59.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fad9b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:59.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fad9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:59.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fada8862640 5 monclient: authenticate success, global_id 4667 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fada8862640 10 monclient: _renew_subs 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fada8862640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fada8862640 1 -- 172.21.15.155:0/4125579339 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada40b0130 con 0x7fada40b2800 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fada8862640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fada8862640 10 monclient: _renew_subs 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fada8862640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fada8862640 1 -- 172.21.15.155:0/4125579339 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fada40ae8e0 con 0x7fada40b2800 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.800+0000 7fada8862640 1 librados: init done 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.802+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad8c001c50 con 0x7fada40b2800 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.802+0000 7fad9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.802+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7fad8c003a40 con 0x7fada40b2800 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.802+0000 7fad9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:59.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.802+0000 7fad9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.802+0000 7fada1d9b640 4 set_mon_vals no callback set 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.802+0000 7fad9b7fe640 10 monclient: dump: 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.802+0000 7fada8862640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:59.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.803+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7fad8c003c60 con 0x7fada40b2800 2024-03-29T11:44:59.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.803+0000 7fada8862640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:59.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.803+0000 7fada8862640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:44:59.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.803+0000 7fada8862640 1 -- 172.21.15.155:0/4125579339 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad88005080 con 0x7fada40b2800 2024-03-29T11:44:59.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.803+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 339942357 0 0) 0x7fad8c084ae0 con 0x7fada40b2800 2024-03-29T11:44:59.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.803+0000 7fad9b7fe640 10 client.4667.objecter ms_dispatch 0x7fada4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:44:59.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.803+0000 7fad9b7fe640 3 client.4667.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:44:59.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.803+0000 7fad9b7fe640 3 client.4667.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:44:59.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.804+0000 7fad9b7fe640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-29T11:44:59.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.804+0000 7fada2d9d640 10 monclient: get_auth_request con 0x7fad8406db50 auth_method 0 2024-03-29T11:44:59.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.804+0000 7fad9b7fe640 10 client.4667.objecter ms_handle_connect 0x7fad8406db50 2024-03-29T11:44:59.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.811+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad8c0503c0 con 0x7fada40b2800 2024-03-29T11:44:59.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.811+0000 7fad9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:59.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:44:59.811+0000 7fad9b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:00.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.133+0000 7fada8862640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:00.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.133+0000 7fada8862640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:00.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.133+0000 7fada8862640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:45:00.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.133+0000 7fada8862640 1 -- 172.21.15.155:0/4125579339 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad88005330 con 0x7fada40b2800 2024-03-29T11:45:00.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.137+0000 7fad9b7fe640 1 -- 172.21.15.155:0/4125579339 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1459448867) 0x7fad8c0522f0 con 0x7fada40b2800 2024-03-29T11:45:00.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.137+0000 7fad9b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:00.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.137+0000 7fad9b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:00.141 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:45:00.141 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:44:36.686112+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:36.686112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:00.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.141+0000 7fada8862640 10 librados: watch_flush enter 2024-03-29T11:45:00.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.143+0000 7fada8862640 10 librados: watch_flush exit 2024-03-29T11:45:00.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.143+0000 7fada8862640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-29T11:45:00.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.143+0000 7fada8862640 10 client.4667.objecter successfully canceled tick 2024-03-29T11:45:00.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.143+0000 7fada8862640 1 -- 172.21.15.155:0/4125579339 >> v1:172.21.15.155:6816/1760814708 conn(0x7fad8406db50 legacy=0x7fad84070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:00.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.143+0000 7fada8862640 10 monclient: shutdown 2024-03-29T11:45:00.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.143+0000 7fada8862640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:00.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.143+0000 7fada8862640 1 -- 172.21.15.155:0/4125579339 >> v1:172.21.15.155:6790/0 conn(0x7fada40b2800 legacy=0x7fada41d7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:00.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.143+0000 7fada8862640 1 -- 172.21.15.155:0/4125579339 shutdown_connections 2024-03-29T11:45:00.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.143+0000 7fada8862640 1 -- 172.21.15.155:0/4125579339 >> 172.21.15.155:0/4125579339 conn(0x7fada4066a00 msgr2=0x7fada4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:00.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.147+0000 7fada8862640 1 -- 172.21.15.155:0/4125579339 shutdown_connections 2024-03-29T11:45:00.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.147+0000 7fada8862640 1 -- 172.21.15.155:0/4125579339 wait complete. 2024-03-29T11:45:00.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:00.147+0000 7fada8862640 1 librados: shutdown 2024-03-29T11:45:00.168 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:06.976 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:07.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.115+0000 7f689bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:45:07.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.115+0000 7f689bfff640 10 monclient: build_initial_monmap 2024-03-29T11:45:07.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.115+0000 7f689bfff640 10 monclient: monmap: 2024-03-29T11:45:07.119 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:45:07.119 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:07.119 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:45:07.116131+0000 2024-03-29T11:45:07.120 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:45:07.116131+0000 2024-03-29T11:45:07.120 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:07.120 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:07.120 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:45:07.120 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:45:07.120 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:45:07.120 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:07.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.116+0000 7f689bfff640 1 Processor -- start 2024-03-29T11:45:07.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.117+0000 7f689bfff640 1 -- start start 2024-03-29T11:45:07.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.117+0000 7f689bfff640 10 monclient: init 2024-03-29T11:45:07.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:07.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689bfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:07.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f689c0b4920 addr v1:172.21.15.183:6789/0 2024-03-29T11:45:07.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f689c0b4200 addr v1:172.21.15.155:6789/0 2024-03-29T11:45:07.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f689c13f9a0 addr v1:172.21.15.155:6790/0 2024-03-29T11:45:07.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689bfff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689c0b0930 con 0x7f689c0b4200 2024-03-29T11:45:07.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689bfff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689c0af460 con 0x7f689c0b4920 2024-03-29T11:45:07.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689bfff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689c0aeda0 con 0x7f689c13f9a0 2024-03-29T11:45:07.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:07.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689bfff640 10 monclient(hunting): authenticate will time out at 1489.244233s 2024-03-29T11:45:07.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689b7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f689c13f9a0 0x7f689c14d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:35420/0 (socket says 172.21.15.155:35420) 2024-03-29T11:45:07.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.118+0000 7f689b7fe640 1 -- 172.21.15.155:0/549366794 learned_addr learned my addr 172.21.15.155:0/549366794 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:45:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 1 -- 172.21.15.155:0/549366794 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700188440 0 0) 0x7f689c0af460 con 0x7f689c0b4920 2024-03-29T11:45:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 1 -- 172.21.15.155:0/549366794 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6888003520 con 0x7f689c0b4920 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 1 -- 172.21.15.155:0/549366794 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638286502 0 0) 0x7f689c0b0930 con 0x7f689c0b4200 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 1 -- 172.21.15.155:0/549366794 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f689c0af460 con 0x7f689c0b4200 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 1 -- 172.21.15.155:0/549366794 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 208729606 0 0) 0x7f6888003520 con 0x7f689c0b4920 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 1 -- 172.21.15.155:0/549366794 >> v1:172.21.15.155:6790/0 conn(0x7f689c13f9a0 legacy=0x7f689c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 1 -- 172.21.15.155:0/549366794 >> v1:172.21.15.155:6789/0 conn(0x7f689c0b4200 legacy=0x7f689c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 1 monclient: found mon.noname-c 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 1 -- 172.21.15.155:0/549366794 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c0af980 con 0x7f689c0b4920 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f689bfff640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f689bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 1 -- 172.21.15.155:0/549366794 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6884001690 con 0x7f689c0b4920 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f6899ffb640 10 monclient: dump: 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.125 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.119+0000 7f689bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.120+0000 7f6899ffb640 1 -- 172.21.15.155:0/549366794 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68840018b0 con 0x7f689c0b4920 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.120+0000 7f6899ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.120+0000 7f689bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.120+0000 7f689bfff640 4 set_mon_vals no callback set 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.120+0000 7f6899ffb640 1 -- 172.21.15.155:0/549366794 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6884001c40 con 0x7f689c0b4920 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.120+0000 7f6899ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.120+0000 7f6899ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:07.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.120+0000 7f6899ffb640 10 monclient: dump: 2024-03-29T11:45:07.127 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:07.127 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:07.127 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.127 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.127 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:07.127 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:07.127 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:07.127 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:07.127 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:07.127 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:07.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.120+0000 7f689bfff640 10 monclient: shutdown 2024-03-29T11:45:07.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.120+0000 7f689bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:07.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.120+0000 7f689bfff640 1 -- 172.21.15.155:0/549366794 >> v1:172.21.15.183:6789/0 conn(0x7f689c0b4920 legacy=0x7f689c13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.121+0000 7f689bfff640 1 -- 172.21.15.155:0/549366794 shutdown_connections 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.121+0000 7f689bfff640 1 -- 172.21.15.155:0/549366794 >> 172.21.15.155:0/549366794 conn(0x7f689c06d230 msgr2=0x7f689c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.121+0000 7f689bfff640 1 -- 172.21.15.155:0/549366794 shutdown_connections 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.121+0000 7f689bfff640 1 -- 172.21.15.155:0/549366794 wait complete. 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.121+0000 7f689bfff640 10 monclient: build_initial_monmap 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.121+0000 7f689bfff640 10 monclient: monmap: 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:45:07.134 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.122+0000 7f689bfff640 1 librados: starting msgr at 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.122+0000 7f689bfff640 1 librados: starting objecter 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.122+0000 7f689bfff640 1 Processor -- start 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.122+0000 7f689bfff640 1 -- start start 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.122+0000 7f689bfff640 1 librados: setting wanted keys 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.122+0000 7f689bfff640 1 librados: calling monclient init 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.122+0000 7f689bfff640 10 monclient: init 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f689c0b4200 addr v1:172.21.15.155:6790/0 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f689c0b4920 addr v1:172.21.15.155:6789/0 2024-03-29T11:45:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f689c13f9a0 addr v1:172.21.15.183:6789/0 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689bfff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689c0ac150 con 0x7f689c0b4920 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689bfff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689c1da3d0 con 0x7f689c13f9a0 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689bfff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689c1db000 con 0x7f689c0b4200 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689bfff640 10 monclient(hunting): authenticate will time out at 1489.249463s 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689a7fc640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f689c0b4920 0x7f689c13a120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:37352/0 (socket says 172.21.15.155:37352) 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.123+0000 7f689a7fc640 1 -- 172.21.15.155:0/214374257 learned_addr learned my addr 172.21.15.155:0/214374257 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 client.?.objecter ms_handle_connect 0x7f689c0b4200 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 client.?.objecter ms_handle_connect 0x7f689c0b4920 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 client.?.objecter ms_handle_connect 0x7f689c13f9a0 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111939231 0 0) 0x7f689c0ac150 con 0x7f689c0b4920 2024-03-29T11:45:07.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f687c004eb0 con 0x7f689c0b4920 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060077684 0 0) 0x7f689c1da3d0 con 0x7f689c13f9a0 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.124+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f689c0ac150 con 0x7f689c13f9a0 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239949064 0 0) 0x7f689c1db000 con 0x7f689c0b4200 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:07.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f689c1da3d0 con 0x7f689c0b4200 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2153776319 0 0) 0x7f689c1da3d0 con 0x7f689c0b4200 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f689c1db000 con 0x7f689c0b4200 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f68840015d0 con 0x7f689c0b4200 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 10 monclient(hunting): dump: 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:07.138 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 923417188 0 0) 0x7f687c004eb0 con 0x7f689c0b4920 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f689c1da3d0 con 0x7f689c0b4920 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2792547411 0 0) 0x7f689c0ac150 con 0x7f689c13f9a0 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f687c004eb0 con 0x7f689c13f9a0 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.125+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f688c001d10 con 0x7f689c0b4920 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 10 monclient(hunting): dump: 2024-03-29T11:45:07.139 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f68940016c0 con 0x7f689c13f9a0 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 10 monclient(hunting): dump: 2024-03-29T11:45:07.140 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 814255409 0 0) 0x7f689c1db000 con 0x7f689c0b4200 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 >> v1:172.21.15.183:6789/0 conn(0x7f689c13f9a0 legacy=0x7f689c0aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 >> v1:172.21.15.155:6789/0 conn(0x7f689c0b4920 legacy=0x7f689c13a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 1 monclient: found mon.c 2024-03-29T11:45:07.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c1db190 con 0x7f689c0b4200 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f68937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f689bfff640 5 monclient: authenticate success, global_id 4673 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f689bfff640 10 monclient: _renew_subs 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f689bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f689bfff640 1 -- 172.21.15.155:0/214374257 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689c140fb0 con 0x7f689c0b4200 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f689bfff640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f689bfff640 10 monclient: _renew_subs 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f689bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:45:07.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f689bfff640 1 -- 172.21.15.155:0/214374257 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f689c13aa10 con 0x7f689c0b4200 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.126+0000 7f689bfff640 1 librados: init done 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.127+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68840011f0 con 0x7f689c0b4200 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.127+0000 7f68937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.127+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f6884001730 con 0x7f689c0b4200 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.127+0000 7f68937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.127+0000 7f68937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.127+0000 7f68937fe640 10 monclient: dump: 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:07.143 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.127+0000 7f68997fa640 4 set_mon_vals no callback set 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.127+0000 7f689bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.128+0000 7f689bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.128+0000 7f689bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.128+0000 7f689bfff640 1 -- 172.21.15.155:0/214374257 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6880005080 con 0x7f689c0b4200 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.129+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f68840198b0 con 0x7f689c0b4200 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.129+0000 7f689a7fc640 10 monclient: get_auth_request con 0x7f687c06e030 auth_method 0 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.129+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 339942357 0 0) 0x7f6884083830 con 0x7f689c0b4200 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.129+0000 7f68937fe640 10 client.4673.objecter ms_dispatch 0x7f689c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.129+0000 7f68937fe640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.129+0000 7f68937fe640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:07.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.129+0000 7f68937fe640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T11:45:07.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.130+0000 7f68937fe640 10 client.4673.objecter ms_handle_connect 0x7f687c06e030 2024-03-29T11:45:07.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.137+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f688404ef40 con 0x7f689c0b4200 2024-03-29T11:45:07.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.137+0000 7f68937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:07.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.137+0000 7f68937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:07.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.486+0000 7f689bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:07.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.486+0000 7f689bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:07.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.486+0000 7f689bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:45:07.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.486+0000 7f689bfff640 1 -- 172.21.15.155:0/214374257 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6880005330 con 0x7f689c0b4200 2024-03-29T11:45:07.490 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:45:07.490 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:44:36.686112+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:36.686112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:07.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.487+0000 7f68937fe640 1 -- 172.21.15.155:0/214374257 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1459448867) 0x7f6884051040 con 0x7f689c0b4200 2024-03-29T11:45:07.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.487+0000 7f68937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:07.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.487+0000 7f68937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:07.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.490+0000 7f689bfff640 10 librados: watch_flush enter 2024-03-29T11:45:07.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.490+0000 7f689bfff640 10 librados: watch_flush exit 2024-03-29T11:45:07.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.490+0000 7f689bfff640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T11:45:07.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.490+0000 7f689bfff640 10 client.4673.objecter successfully canceled tick 2024-03-29T11:45:07.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.490+0000 7f689bfff640 1 -- 172.21.15.155:0/214374257 >> v1:172.21.15.155:6816/1760814708 conn(0x7f687c06e030 legacy=0x7f687c0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:07.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.490+0000 7f689bfff640 10 monclient: shutdown 2024-03-29T11:45:07.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.490+0000 7f689bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:07.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.490+0000 7f689bfff640 1 -- 172.21.15.155:0/214374257 >> v1:172.21.15.155:6790/0 conn(0x7f689c0b4200 legacy=0x7f689c1da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:07.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.490+0000 7f689bfff640 1 -- 172.21.15.155:0/214374257 shutdown_connections 2024-03-29T11:45:07.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.490+0000 7f689bfff640 1 -- 172.21.15.155:0/214374257 >> 172.21.15.155:0/214374257 conn(0x7f689c06d230 msgr2=0x7f689c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:07.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.490+0000 7f689bfff640 1 -- 172.21.15.155:0/214374257 shutdown_connections 2024-03-29T11:45:07.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.491+0000 7f689bfff640 1 -- 172.21.15.155:0/214374257 wait complete. 2024-03-29T11:45:07.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:07.491+0000 7f689bfff640 1 librados: shutdown 2024-03-29T11:45:07.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:14.316 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:14.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.477+0000 7f24bea4d640 10 monclient: get_monmap_and_config 2024-03-29T11:45:14.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.477+0000 7f24bea4d640 10 monclient: build_initial_monmap 2024-03-29T11:45:14.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.477+0000 7f24bea4d640 10 monclient: monmap: 2024-03-29T11:45:14.481 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:45:14.481 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:14.482 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:45:14.477512+0000 2024-03-29T11:45:14.482 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:45:14.477512+0000 2024-03-29T11:45:14.482 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:14.482 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:14.482 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:45:14.482 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:45:14.482 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:45:14.482 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:14.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.480+0000 7f24bea4d640 1 Processor -- start 2024-03-29T11:45:14.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.481+0000 7f24bea4d640 1 -- start start 2024-03-29T11:45:14.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.481+0000 7f24bea4d640 10 monclient: init 2024-03-29T11:45:14.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.482+0000 7f24bea4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:14.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.482+0000 7f24bea4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:14.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.482+0000 7f24bea4d640 10 monclient(hunting): picked mon.noname-b con 0x7f24b0006890 addr v1:172.21.15.155:6790/0 2024-03-29T11:45:14.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.482+0000 7f24bea4d640 10 monclient(hunting): picked mon.noname-a con 0x7f24b0006c70 addr v1:172.21.15.155:6789/0 2024-03-29T11:45:14.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.482+0000 7f24bea4d640 10 monclient(hunting): picked mon.noname-c con 0x7f24b002b4c0 addr v1:172.21.15.183:6789/0 2024-03-29T11:45:14.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.482+0000 7f24bea4d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b002c830 con 0x7f24b0006c70 2024-03-29T11:45:14.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.482+0000 7f24bea4d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b002afa0 con 0x7f24b002b4c0 2024-03-29T11:45:14.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.482+0000 7f24bea4d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b002aba0 con 0x7f24b0006890 2024-03-29T11:45:14.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.482+0000 7f24bea4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:14.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.482+0000 7f24bea4d640 10 monclient(hunting): authenticate will time out at 1496.608060s 2024-03-29T11:45:14.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.486+0000 7f24be24c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f24b002b4c0 0x7f24b00cf0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:48739/0 (socket says 172.21.15.155:48739) 2024-03-29T11:45:14.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.486+0000 7f24be24c640 1 -- 172.21.15.155:0/2899555539 learned_addr learned my addr 172.21.15.155:0/2899555539 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:45:14.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.487+0000 7f24bca49640 1 -- 172.21.15.155:0/2899555539 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531409828 0 0) 0x7f24b002afa0 con 0x7f24b002b4c0 2024-03-29T11:45:14.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.487+0000 7f24bca49640 10 monclient(hunting): _negotiate 2024-03-29T11:45:14.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.487+0000 7f24bca49640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:14.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.487+0000 7f24bca49640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:14.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.487+0000 7f24bca49640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:45:14.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.487+0000 7f24bca49640 1 -- 172.21.15.155:0/2899555539 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a4003520 con 0x7f24b002b4c0 2024-03-29T11:45:14.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.488+0000 7f24bca49640 1 -- 172.21.15.155:0/2899555539 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1334808315 0 0) 0x7f24a4003520 con 0x7f24b002b4c0 2024-03-29T11:45:14.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.488+0000 7f24bca49640 1 -- 172.21.15.155:0/2899555539 >> v1:172.21.15.155:6790/0 conn(0x7f24b0006890 legacy=0x7f24b00c0a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:14.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 1 -- 172.21.15.155:0/2899555539 >> v1:172.21.15.155:6789/0 conn(0x7f24b0006c70 legacy=0x7f24b00c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:14.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 10 monclient: _finish_hunting 0 2024-03-29T11:45:14.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 1 monclient: found mon.noname-c 2024-03-29T11:45:14.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:14.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:45:14.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 1 -- 172.21.15.155:0/2899555539 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b002ad30 con 0x7f24b002b4c0 2024-03-29T11:45:14.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 10 monclient: _finish_auth 0 2024-03-29T11:45:14.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 10 monclient: _check_auth_tickets 2024-03-29T11:45:14.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:14.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 1 -- 172.21.15.155:0/2899555539 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f24a8001690 con 0x7f24b002b4c0 2024-03-29T11:45:14.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bea4d640 5 monclient: authenticate success, global_id 4552 2024-03-29T11:45:14.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bea4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:14.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:14.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:14.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 10 monclient: dump: 2024-03-29T11:45:14.497 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:14.498 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:14.498 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:14.498 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:14.498 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:14.498 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:14.498 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:14.499 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:14.499 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:14.499 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:14.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bea4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:14.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 1 -- 172.21.15.155:0/2899555539 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24a80018b0 con 0x7f24b002b4c0 2024-03-29T11:45:14.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:14.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 1 -- 172.21.15.155:0/2899555539 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f24a8001c40 con 0x7f24b002b4c0 2024-03-29T11:45:14.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bea4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:14.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bea4d640 4 set_mon_vals no callback set 2024-03-29T11:45:14.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:14.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:14.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.489+0000 7f24bca49640 10 monclient: dump: 2024-03-29T11:45:14.501 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:14.501 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:14.501 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:14.501 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:14.501 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:14.502 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:14.502 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:14.502 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:14.502 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:14.502 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:14.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.490+0000 7f24bea4d640 10 monclient: shutdown 2024-03-29T11:45:14.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.490+0000 7f24bea4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:14.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.490+0000 7f24bea4d640 1 -- 172.21.15.155:0/2899555539 >> v1:172.21.15.183:6789/0 conn(0x7f24b002b4c0 legacy=0x7f24b00cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:14.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.491+0000 7f24bea4d640 1 -- 172.21.15.155:0/2899555539 shutdown_connections 2024-03-29T11:45:14.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.491+0000 7f24bea4d640 1 -- 172.21.15.155:0/2899555539 >> 172.21.15.155:0/2899555539 conn(0x7f24b00058f0 msgr2=0x7f24b0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:14.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.492+0000 7f24bea4d640 1 -- 172.21.15.155:0/2899555539 shutdown_connections 2024-03-29T11:45:14.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.493+0000 7f24bea4d640 1 -- 172.21.15.155:0/2899555539 wait complete. 2024-03-29T11:45:14.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.494+0000 7f24bea4d640 10 monclient: build_initial_monmap 2024-03-29T11:45:14.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.494+0000 7f24bea4d640 10 monclient: monmap: 2024-03-29T11:45:14.509 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:45:14.510 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:14.510 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:45:14.510 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:45:14.510 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:14.510 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:14.510 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:45:14.510 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:45:14.511 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:45:14.511 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:14.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.495+0000 7f24bea4d640 1 librados: starting msgr at 2024-03-29T11:45:14.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.495+0000 7f24bea4d640 1 librados: starting objecter 2024-03-29T11:45:14.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.495+0000 7f24bea4d640 1 Processor -- start 2024-03-29T11:45:14.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.495+0000 7f24bea4d640 1 -- start start 2024-03-29T11:45:14.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.495+0000 7f24bea4d640 1 librados: setting wanted keys 2024-03-29T11:45:14.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.496+0000 7f24bea4d640 1 librados: calling monclient init 2024-03-29T11:45:14.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.496+0000 7f24bea4d640 10 monclient: init 2024-03-29T11:45:14.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.497+0000 7f24bea4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:14.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.497+0000 7f24bea4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:14.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.497+0000 7f24bea4d640 10 monclient(hunting): picked mon.noname-a con 0x7f24b0006890 addr v1:172.21.15.155:6789/0 2024-03-29T11:45:14.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.497+0000 7f24bea4d640 10 monclient(hunting): picked mon.noname-b con 0x7f24b002b4c0 addr v1:172.21.15.183:6789/0 2024-03-29T11:45:14.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.498+0000 7f24bea4d640 10 monclient(hunting): picked mon.noname-c con 0x7f24b0158540 addr v1:172.21.15.155:6790/0 2024-03-29T11:45:14.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.498+0000 7f24bea4d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b002ce10 con 0x7f24b0006890 2024-03-29T11:45:14.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.498+0000 7f24bea4d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b002de10 con 0x7f24b002b4c0 2024-03-29T11:45:14.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.498+0000 7f24bea4d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b0158a60 con 0x7f24b0158540 2024-03-29T11:45:14.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.498+0000 7f24bea4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:14.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.498+0000 7f24bea4d640 10 monclient(hunting): authenticate will time out at 1496.624692s 2024-03-29T11:45:14.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.500+0000 7f24bda4b640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f24b0006890 0x7f24b00c15c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:43111/0 (socket says 172.21.15.155:43111) 2024-03-29T11:45:14.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.500+0000 7f24bda4b640 1 -- 172.21.15.155:0/4098495989 learned_addr learned my addr 172.21.15.155:0/4098495989 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:45:14.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.502+0000 7f24ae7fc640 10 client.?.objecter ms_handle_connect 0x7f24b0006890 2024-03-29T11:45:14.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.502+0000 7f24ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:14.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.504+0000 7f24ae7fc640 10 client.?.objecter ms_handle_connect 0x7f24b002b4c0 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.504+0000 7f24ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 client.?.objecter ms_handle_connect 0x7f24b0158540 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453541890 0 0) 0x7f24b002de10 con 0x7f24b002b4c0 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c004eb0 con 0x7f24b002b4c0 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618535562 0 0) 0x7f24b40011a0 con 0x7f24b0006890 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:14.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:14.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:14.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:45:14.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b002de10 con 0x7f24b0006890 2024-03-29T11:45:14.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168779460 0 0) 0x7f24b0158a60 con 0x7f24b0158540 2024-03-29T11:45:14.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:14.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:14.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:14.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:45:14.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.505+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b40011a0 con 0x7f24b0158540 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154831040 0 0) 0x7f249c004eb0 con 0x7f24b002b4c0 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24b0158a60 con 0x7f24b002b4c0 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f24b804fdd0 con 0x7f24b002b4c0 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 10 monclient(hunting): dump: 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:14.517 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 91161592 0 0) 0x7f24b40011a0 con 0x7f24b0158540 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f249c004eb0 con 0x7f24b0158540 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f24a80013b0 con 0x7f24b0158540 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 10 monclient(hunting): dump: 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:14.518 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 607954109 0 0) 0x7f24b0158a60 con 0x7f24b002b4c0 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 >> v1:172.21.15.155:6790/0 conn(0x7f24b0158540 legacy=0x7f24b0163d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 >> v1:172.21.15.155:6789/0 conn(0x7f24b0006890 legacy=0x7f24b00c15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:14.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 1 monclient: found mon.b 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b00bc510 con 0x7f24b002b4c0 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24bea4d640 5 monclient: authenticate success, global_id 4555 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24bea4d640 10 monclient: _renew_subs 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24bea4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24bea4d640 1 -- 172.21.15.155:0/4098495989 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b00bc760 con 0x7f24b002b4c0 2024-03-29T11:45:14.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24bea4d640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24bea4d640 10 monclient: _renew_subs 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24bea4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24bea4d640 1 -- 172.21.15.155:0/4098495989 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b0031590 con 0x7f24b002b4c0 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.506+0000 7f24bea4d640 1 librados: init done 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.508+0000 7f24a3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.508+0000 7f24a3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.508+0000 7f24a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.508+0000 7f24a3fff640 1 -- 172.21.15.155:0/4098495989 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2498005080 con 0x7f24b002b4c0 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.509+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24b804f400 con 0x7f24b002b4c0 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.509+0000 7f24ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.509+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f24b8075f10 con 0x7f24b002b4c0 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.509+0000 7f24af7fe640 4 set_mon_vals no callback set 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.510+0000 7f24ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:14.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.510+0000 7f24ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:14.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.510+0000 7f24ae7fc640 10 monclient: dump: 2024-03-29T11:45:14.522 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:14.522 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:14.522 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:14.522 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:14.522 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:14.522 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:14.522 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:14.522 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:14.522 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:14.523 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:14.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.513+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f24b8067bf0 con 0x7f24b002b4c0 2024-03-29T11:45:14.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.514+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 339942357 0 0) 0x7f24b80fe100 con 0x7f24b002b4c0 2024-03-29T11:45:14.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.515+0000 7f24bda4b640 10 monclient: get_auth_request con 0x7f249c06dc40 auth_method 0 2024-03-29T11:45:14.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.517+0000 7f24ae7fc640 10 client.4555.objecter ms_dispatch 0x7f24b8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:14.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.517+0000 7f24ae7fc640 3 client.4555.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:14.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.517+0000 7f24ae7fc640 3 client.4555.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:14.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.518+0000 7f24ae7fc640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-29T11:45:14.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.518+0000 7f24ae7fc640 10 client.4555.objecter ms_handle_connect 0x7f249c06dc40 2024-03-29T11:45:14.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.518+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24b80c99e0 con 0x7f24b002b4c0 2024-03-29T11:45:14.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.518+0000 7f24ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:14.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.518+0000 7f24ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.851+0000 7f24a3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.851+0000 7f24a3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.851+0000 7f24a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.851+0000 7f24a3fff640 1 -- 172.21.15.155:0/4098495989 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2498005330 con 0x7f24b002b4c0 2024-03-29T11:45:14.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.853+0000 7f24ae7fc640 1 -- 172.21.15.155:0/4098495989 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1459448867) 0x7f24b80cb910 con 0x7f24b002b4c0 2024-03-29T11:45:14.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.853+0000 7f24ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:14.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.853+0000 7f24ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:14.857 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:45:14.858 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:44:36.686112+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:36.686112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:14.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.859+0000 7f24a3fff640 10 librados: watch_flush enter 2024-03-29T11:45:14.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.859+0000 7f24a3fff640 10 librados: watch_flush exit 2024-03-29T11:45:14.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.859+0000 7f24a3fff640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-29T11:45:14.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.859+0000 7f24a3fff640 10 client.4555.objecter successfully canceled tick 2024-03-29T11:45:14.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.860+0000 7f24a3fff640 1 -- 172.21.15.155:0/4098495989 >> v1:172.21.15.155:6816/1760814708 conn(0x7f249c06dc40 legacy=0x7f249c070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:14.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.860+0000 7f24a3fff640 10 monclient: shutdown 2024-03-29T11:45:14.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.860+0000 7f24a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:14.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.860+0000 7f24a3fff640 1 -- 172.21.15.155:0/4098495989 >> v1:172.21.15.183:6789/0 conn(0x7f24b002b4c0 legacy=0x7f24b00bbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:14.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.860+0000 7f24a3fff640 1 -- 172.21.15.155:0/4098495989 shutdown_connections 2024-03-29T11:45:14.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.860+0000 7f24a3fff640 1 -- 172.21.15.155:0/4098495989 >> 172.21.15.155:0/4098495989 conn(0x7f24b00058f0 msgr2=0x7f24b0038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:14.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.860+0000 7f24a3fff640 1 -- 172.21.15.155:0/4098495989 shutdown_connections 2024-03-29T11:45:14.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.860+0000 7f24a3fff640 1 -- 172.21.15.155:0/4098495989 wait complete. 2024-03-29T11:45:14.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:14.861+0000 7f24a3fff640 1 librados: shutdown 2024-03-29T11:45:14.885 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:21.693 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:21.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.840+0000 7f3f3b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.840+0000 7f3f3b59e640 10 monclient: build_initial_monmap 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.840+0000 7f3f3b59e640 10 monclient: monmap: 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:45:21.842334+0000 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:45:21.842334+0000 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:45:21.846 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:21.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.843+0000 7f3f3b59e640 1 Processor -- start 2024-03-29T11:45:21.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.843+0000 7f3f3b59e640 1 -- start start 2024-03-29T11:45:21.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.843+0000 7f3f3b59e640 10 monclient: init 2024-03-29T11:45:21.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.845+0000 7f3f3b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:21.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.845+0000 7f3f3b59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:21.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.845+0000 7f3f3b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f3f3c06ade0 addr v1:172.21.15.155:6789/0 2024-03-29T11:45:21.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.845+0000 7f3f3b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3c070b00 addr v1:172.21.15.183:6789/0 2024-03-29T11:45:21.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.848+0000 7f3f3a59c640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f3f3c06ade0 0x7f3f3c140460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48703/0 (socket says 172.21.15.155:48703) 2024-03-29T11:45:21.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.848+0000 7f3f3b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f3f3c1496d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:45:21.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.848+0000 7f3f3ad9d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f3f3c1496d0 0x7f3f3c14bac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:52311/0 (socket says 172.21.15.155:52311) 2024-03-29T11:45:21.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.848+0000 7f3f3ad9d640 1 -- 172.21.15.155:0/848834351 learned_addr learned my addr 172.21.15.155:0/848834351 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:45:21.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.848+0000 7f3f3b59e640 1 -- 172.21.15.155:0/848834351 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0b0600 con 0x7f3f3c06ade0 2024-03-29T11:45:21.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.849+0000 7f3f3b59e640 1 -- 172.21.15.155:0/848834351 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0b08e0 con 0x7f3f3c070b00 2024-03-29T11:45:21.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.849+0000 7f3f3b59e640 1 -- 172.21.15.155:0/848834351 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c072520 con 0x7f3f3c1496d0 2024-03-29T11:45:21.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.849+0000 7f3f3b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:21.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.849+0000 7f3f3b59e640 10 monclient(hunting): authenticate will time out at 1503.976241s 2024-03-29T11:45:21.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.852+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824191662 0 0) 0x7f3f3c0b08e0 con 0x7f3f3c070b00 2024-03-29T11:45:21.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.852+0000 7f3f3959a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:21.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.852+0000 7f3f3959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.852+0000 7f3f3959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.852+0000 7f3f3959a640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.852+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f24003520 con 0x7f3f3c070b00 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.852+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851274075 0 0) 0x7f3f3c072520 con 0x7f3f3c1496d0 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.852+0000 7f3f3959a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f3c0b08e0 con 0x7f3f3c1496d0 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556637494 0 0) 0x7f3f3c0b0600 con 0x7f3f3c06ade0 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:21.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f3c072520 con 0x7f3f3c06ade0 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2548467219 0 0) 0x7f3f24003520 con 0x7f3f3c070b00 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 >> v1:172.21.15.155:6790/0 conn(0x7f3f3c1496d0 legacy=0x7f3f3c14bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 >> v1:172.21.15.155:6789/0 conn(0x7f3f3c06ade0 legacy=0x7f3f3c140460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient: _finish_hunting 0 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 1 monclient: found mon.noname-c 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c13ff40 con 0x7f3f3c070b00 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient: _finish_auth 0 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient: _check_auth_tickets 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3b59e640 5 monclient: authenticate success, global_id 4558 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:21.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3f34001990 con 0x7f3f3c070b00 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient: dump: 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:21.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f34001af0 con 0x7f3f3c070b00 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.853+0000 7f3f3959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.854+0000 7f3f3b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.854+0000 7f3f3b59e640 4 set_mon_vals no callback set 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.854+0000 7f3f3959a640 1 -- 172.21.15.155:0/848834351 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3f34001f40 con 0x7f3f3c070b00 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.854+0000 7f3f3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.854+0000 7f3f3959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.854+0000 7f3f3959a640 10 monclient: dump: 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:21.863 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:21.864 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:21.864 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:21.864 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:21.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.854+0000 7f3f3b59e640 10 monclient: shutdown 2024-03-29T11:45:21.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.854+0000 7f3f3b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:21.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.854+0000 7f3f3b59e640 1 -- 172.21.15.155:0/848834351 >> v1:172.21.15.183:6789/0 conn(0x7f3f3c070b00 legacy=0x7f3f3c13cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:21.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.858+0000 7f3f3b59e640 1 -- 172.21.15.155:0/848834351 shutdown_connections 2024-03-29T11:45:21.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.858+0000 7f3f3b59e640 1 -- 172.21.15.155:0/848834351 >> 172.21.15.155:0/848834351 conn(0x7f3f3c067420 msgr2=0x7f3f3c0b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:21.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.860+0000 7f3f3b59e640 1 -- 172.21.15.155:0/848834351 shutdown_connections 2024-03-29T11:45:21.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.860+0000 7f3f3b59e640 1 -- 172.21.15.155:0/848834351 wait complete. 2024-03-29T11:45:21.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.864+0000 7f3f3b59e640 10 monclient: build_initial_monmap 2024-03-29T11:45:21.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.864+0000 7f3f3b59e640 10 monclient: monmap: 2024-03-29T11:45:21.869 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:45:21.869 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:21.869 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:45:21.869 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:45:21.869 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:21.869 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:21.869 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:45:21.869 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:45:21.869 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:45:21.870 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:21.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.865+0000 7f3f3b59e640 1 librados: starting msgr at 2024-03-29T11:45:21.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.865+0000 7f3f3b59e640 1 librados: starting objecter 2024-03-29T11:45:21.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.865+0000 7f3f3b59e640 1 Processor -- start 2024-03-29T11:45:21.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.867+0000 7f3f3b59e640 1 -- start start 2024-03-29T11:45:21.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.867+0000 7f3f3b59e640 1 librados: setting wanted keys 2024-03-29T11:45:21.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.867+0000 7f3f3b59e640 1 librados: calling monclient init 2024-03-29T11:45:21.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.867+0000 7f3f3b59e640 10 monclient: init 2024-03-29T11:45:21.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.869+0000 7f3f3b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:21.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.869+0000 7f3f3b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:21.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.872+0000 7f3f3b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f3f3c06ade0 addr v1:172.21.15.183:6789/0 2024-03-29T11:45:21.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.872+0000 7f3f3a59c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f3f3c06ade0 0x7f3f3c1431f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:43645/0 (socket says 172.21.15.155:43645) 2024-03-29T11:45:21.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.872+0000 7f3f3a59c640 1 -- 172.21.15.155:0/3164211763 learned_addr learned my addr 172.21.15.155:0/3164211763 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:45:21.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.872+0000 7f3f3b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f3f3c070b00 addr v1:172.21.15.155:6789/0 2024-03-29T11:45:21.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.873+0000 7f3f2affd640 10 client.?.objecter ms_handle_connect 0x7f3f3c06ade0 2024-03-29T11:45:21.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.873+0000 7f3f2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:21.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.874+0000 7f3f3b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3c1496d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:45:21.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.874+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c137df0 con 0x7f3f3c070b00 2024-03-29T11:45:21.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.874+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c142cd0 con 0x7f3f3c06ade0 2024-03-29T11:45:21.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.874+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c144c60 con 0x7f3f3c1496d0 2024-03-29T11:45:21.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.874+0000 7f3f3b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:21.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.874+0000 7f3f3b59e640 10 monclient(hunting): authenticate will time out at 1504.001856s 2024-03-29T11:45:21.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.875+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145337161 0 0) 0x7f3f3c142cd0 con 0x7f3f3c06ade0 2024-03-29T11:45:21.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.875+0000 7f3f2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:21.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.875+0000 7f3f2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:21.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.875+0000 7f3f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:21.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.875+0000 7f3f2affd640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:45:21.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.875+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1c004eb0 con 0x7f3f3c06ade0 2024-03-29T11:45:21.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.876+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2468128015 0 0) 0x7f3f1c004eb0 con 0x7f3f3c06ade0 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.876+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f3c142cd0 con 0x7f3f3c06ade0 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.876+0000 7f3f2affd640 10 client.?.objecter ms_handle_connect 0x7f3f3c1496d0 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.876+0000 7f3f2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3f2c0019c0 con 0x7f3f3c06ade0 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 10 monclient(hunting): dump: 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:21.885 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561965699 0 0) 0x7f3f3c144c60 con 0x7f3f3c1496d0 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 10 monclient(hunting): my global_id is 4682 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1c004eb0 con 0x7f3f3c1496d0 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2436553681 0 0) 0x7f3f3c142cd0 con 0x7f3f3c06ade0 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 >> v1:172.21.15.155:6790/0 conn(0x7f3f3c1496d0 legacy=0x7f3f3c1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 >> v1:172.21.15.155:6789/0 conn(0x7f3f3c070b00 legacy=0x7f3f3c136fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:21.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.877+0000 7f3f2affd640 1 monclient: found mon.b 2024-03-29T11:45:21.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:21.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f2affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:21.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c144f40 con 0x7f3f3c06ade0 2024-03-29T11:45:21.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f2affd640 10 monclient: _finish_auth 0 2024-03-29T11:45:21.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f2affd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:21.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:21.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f2c001b20 con 0x7f3f3c06ade0 2024-03-29T11:45:21.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f3b59e640 5 monclient: authenticate success, global_id 4561 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f3b59e640 10 monclient: _renew_subs 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f3b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c143900 con 0x7f3f3c06ade0 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f3b59e640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f3b59e640 10 monclient: _renew_subs 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.878+0000 7f3f3b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.879+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c141080 con 0x7f3f3c06ade0 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.879+0000 7f3f3b59e640 1 librados: init done 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.879+0000 7f3f2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.879+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f3f2c004c30 con 0x7f3f3c06ade0 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.879+0000 7f3f2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.879+0000 7f3f2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.879+0000 7f3f38d99640 4 set_mon_vals no callback set 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.879+0000 7f3f2affd640 10 monclient: dump: 2024-03-29T11:45:21.888 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:21.889 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:21.889 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:21.889 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:21.889 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:21.889 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:21.889 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:21.889 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:21.889 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:21.889 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:21.890 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.880+0000 7f3f3b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:21.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.880+0000 7f3f3b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:21.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.880+0000 7f3f3b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:21.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.881+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f3f2c004d90 con 0x7f3f3c06ade0 2024-03-29T11:45:21.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.882+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 339942357 0 0) 0x7f3f2c084b70 con 0x7f3f3c06ade0 2024-03-29T11:45:21.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.882+0000 7f3f2affd640 10 client.4561.objecter ms_dispatch 0x7f3f3c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:21.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.882+0000 7f3f2affd640 3 client.4561.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:21.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.882+0000 7f3f2affd640 3 client.4561.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:21.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.882+0000 7f3f2affd640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-29T11:45:21.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.881+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f20005080 con 0x7f3f3c06ade0 2024-03-29T11:45:21.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.882+0000 7f3f39d9b640 10 monclient: get_auth_request con 0x7f3f1c06de40 auth_method 0 2024-03-29T11:45:21.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.883+0000 7f3f2affd640 10 client.4561.objecter ms_handle_connect 0x7f3f1c06de40 2024-03-29T11:45:21.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.889+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f2c050450 con 0x7f3f3c06ade0 2024-03-29T11:45:21.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.889+0000 7f3f2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:21.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:21.889+0000 7f3f2affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.224+0000 7f3f3b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.224+0000 7f3f3b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.224+0000 7f3f3b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.224+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f20005330 con 0x7f3f3c06ade0 2024-03-29T11:45:22.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.226+0000 7f3f2affd640 1 -- 172.21.15.155:0/3164211763 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1459448867) 0x7f3f2c052380 con 0x7f3f3c06ade0 2024-03-29T11:45:22.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.227+0000 7f3f2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:22.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.227+0000 7f3f2affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:22.232 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:45:22.232 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:44:36.686112+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:36.686112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:22.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.231+0000 7f3f3b59e640 10 librados: watch_flush enter 2024-03-29T11:45:22.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.231+0000 7f3f3b59e640 10 librados: watch_flush exit 2024-03-29T11:45:22.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.231+0000 7f3f3b59e640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-29T11:45:22.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.231+0000 7f3f3b59e640 10 client.4561.objecter successfully canceled tick 2024-03-29T11:45:22.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.231+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 >> v1:172.21.15.155:6816/1760814708 conn(0x7f3f1c06de40 legacy=0x7f3f1c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:22.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.231+0000 7f3f3b59e640 10 monclient: shutdown 2024-03-29T11:45:22.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.231+0000 7f3f3b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:22.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.231+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 >> v1:172.21.15.183:6789/0 conn(0x7f3f3c06ade0 legacy=0x7f3f3c1431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:22.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.232+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 shutdown_connections 2024-03-29T11:45:22.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.232+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 >> 172.21.15.155:0/3164211763 conn(0x7f3f3c067420 msgr2=0x7f3f3c0b4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:22.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.232+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 shutdown_connections 2024-03-29T11:45:22.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.232+0000 7f3f3b59e640 1 -- 172.21.15.155:0/3164211763 wait complete. 2024-03-29T11:45:22.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:22.232+0000 7f3f3b59e640 1 librados: shutdown 2024-03-29T11:45:22.254 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:29.061 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:29.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.187+0000 7f122e98d640 10 monclient: get_monmap_and_config 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.187+0000 7f122e98d640 10 monclient: build_initial_monmap 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.187+0000 7f122e98d640 10 monclient: monmap: 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:45:29.189233+0000 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:45:29.189233+0000 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:45:29.193 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:29.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.188+0000 7f122e98d640 1 Processor -- start 2024-03-29T11:45:29.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.189+0000 7f122e98d640 1 -- start start 2024-03-29T11:45:29.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.189+0000 7f122e98d640 10 monclient: init 2024-03-29T11:45:29.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.189+0000 7f122e98d640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:29.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f122e98d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:29.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f122e98d640 10 monclient(hunting): picked mon.noname-c con 0x7f12280b0770 addr v1:172.21.15.183:6789/0 2024-03-29T11:45:29.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f122e98d640 10 monclient(hunting): picked mon.noname-b con 0x7f122814a700 addr v1:172.21.15.155:6790/0 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f122e98d640 10 monclient(hunting): picked mon.noname-a con 0x7f12280ac5a0 addr v1:172.21.15.155:6789/0 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f122e98d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12280aba70 con 0x7f12280ac5a0 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f122e98d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12280aecb0 con 0x7f12280b0770 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f122e98d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12280af150 con 0x7f122814a700 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f122e98d640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f122e98d640 10 monclient(hunting): authenticate will time out at 1511.316967s 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f121f7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f122814a700 0x7f12280abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:56652/0 (socket says 172.21.15.155:56652) 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f121f7fe640 1 -- 172.21.15.155:0/1964391508 learned_addr learned my addr 172.21.15.155:0/1964391508 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f12277fe640 1 -- 172.21.15.155:0/1964391508 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025870911 0 0) 0x7f12280af150 con 0x7f122814a700 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f12277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f12277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f12277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f12277fe640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:45:29.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f12277fe640 1 -- 172.21.15.155:0/1964391508 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1214003520 con 0x7f122814a700 2024-03-29T11:45:29.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f12277fe640 1 -- 172.21.15.155:0/1964391508 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2089010629 0 0) 0x7f1214003520 con 0x7f122814a700 2024-03-29T11:45:29.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f12277fe640 1 -- 172.21.15.155:0/1964391508 >> v1:172.21.15.183:6789/0 conn(0x7f12280b0770 legacy=0x7f1228149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:29.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.190+0000 7f12277fe640 1 -- 172.21.15.155:0/1964391508 >> v1:172.21.15.155:6789/0 conn(0x7f12280ac5a0 legacy=0x7f1228151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 1 monclient: found mon.noname-b 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 1 -- 172.21.15.155:0/1964391508 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12280ac9d0 con 0x7f122814a700 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f122e98d640 5 monclient: authenticate success, global_id 4685 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f122e98d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 1 -- 172.21.15.155:0/1964391508 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1210001690 con 0x7f122814a700 2024-03-29T11:45:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 10 monclient: dump: 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f122e98d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 1 -- 172.21.15.155:0/1964391508 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12100019a0 con 0x7f122814a700 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f122e98d640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f122e98d640 4 set_mon_vals no callback set 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 1 -- 172.21.15.155:0/1964391508 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1210001d30 con 0x7f122814a700 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.193+0000 7f12277fe640 10 monclient: dump: 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:29.202 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.194+0000 7f122e98d640 10 monclient: shutdown 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.194+0000 7f122e98d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.194+0000 7f122e98d640 1 -- 172.21.15.155:0/1964391508 >> v1:172.21.15.155:6790/0 conn(0x7f122814a700 legacy=0x7f12280abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.194+0000 7f122e98d640 1 -- 172.21.15.155:0/1964391508 shutdown_connections 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.194+0000 7f122e98d640 1 -- 172.21.15.155:0/1964391508 >> 172.21.15.155:0/1964391508 conn(0x7f122806b660 msgr2=0x7f122806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.194+0000 7f122e98d640 1 -- 172.21.15.155:0/1964391508 shutdown_connections 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.194+0000 7f122e98d640 1 -- 172.21.15.155:0/1964391508 wait complete. 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.195+0000 7f122e98d640 10 monclient: build_initial_monmap 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.195+0000 7f122e98d640 10 monclient: monmap: 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:29.203 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:45:29.204 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:45:29.204 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:29.204 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:29.204 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:45:29.204 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:45:29.204 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:45:29.204 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:29.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.195+0000 7f122e98d640 1 librados: starting msgr at 2024-03-29T11:45:29.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.195+0000 7f122e98d640 1 librados: starting objecter 2024-03-29T11:45:29.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.195+0000 7f122e98d640 1 Processor -- start 2024-03-29T11:45:29.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 1 -- start start 2024-03-29T11:45:29.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 1 librados: setting wanted keys 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 1 librados: calling monclient init 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 10 monclient: init 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 10 monclient(hunting): picked mon.noname-a con 0x7f12280ac5a0 addr v1:172.21.15.155:6789/0 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 10 monclient(hunting): picked mon.noname-c con 0x7f12280b0770 addr v1:172.21.15.155:6790/0 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 10 monclient(hunting): picked mon.noname-b con 0x7f122814a700 addr v1:172.21.15.183:6789/0 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1228128280 con 0x7f12280ac5a0 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1228129710 con 0x7f122814a700 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1228128ae0 con 0x7f12280b0770 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.196+0000 7f122e98d640 10 monclient(hunting): authenticate will time out at 1511.323824s 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.197+0000 7f122cf2a640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f122814a700 0x7f12281e7100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:42430/0 (socket says 172.21.15.155:42430) 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.197+0000 7f122cf2a640 1 -- 172.21.15.155:0/65047148 learned_addr learned my addr 172.21.15.155:0/65047148 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.197+0000 7f12257fa640 10 client.?.objecter ms_handle_connect 0x7f122814a700 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.197+0000 7f12257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.197+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699273266 0 0) 0x7f1228129710 con 0x7f122814a700 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.197+0000 7f12257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.197+0000 7f12257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.197+0000 7f12257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.197+0000 7f12257fa640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.197+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1208004eb0 con 0x7f122814a700 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.197+0000 7f12257fa640 10 client.?.objecter ms_handle_connect 0x7f12280b0770 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2753931243 0 0) 0x7f1208004eb0 con 0x7f122814a700 2024-03-29T11:45:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1228129710 con 0x7f122814a700 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1220001c60 con 0x7f122814a700 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 10 monclient(hunting): dump: 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:29.213 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136916937 0 0) 0x7f1228128ae0 con 0x7f12280b0770 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 10 monclient(hunting): my global_id is 4688 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1208004eb0 con 0x7f12280b0770 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4087137592 0 0) 0x7f1228129710 con 0x7f122814a700 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 >> v1:172.21.15.155:6790/0 conn(0x7f12280b0770 legacy=0x7f12281471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.198+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 >> v1:172.21.15.155:6789/0 conn(0x7f12280ac5a0 legacy=0x7f1228129000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 10 monclient: _finish_hunting 0 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 1 monclient: found mon.b 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12281dfa10 con 0x7f122814a700 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 10 monclient: _finish_auth 0 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 10 monclient: _check_auth_tickets 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f122e98d640 5 monclient: authenticate success, global_id 4564 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f122e98d640 10 monclient: _renew_subs 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f122e98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f122e98d640 1 -- 172.21.15.155:0/65047148 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12281dfc60 con 0x7f122814a700 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f122e98d640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f122e98d640 10 monclient: _renew_subs 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f122e98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f122e98d640 1 -- 172.21.15.155:0/65047148 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1228125c30 con 0x7f122814a700 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f122e98d640 1 librados: init done 2024-03-29T11:45:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1220003970 con 0x7f122814a700 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 66209858 0 0) 0x7f1220001dc0 con 0x7f122814a700 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f12257fa640 10 monclient: dump: 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:fsid 3bbabebe-1eb5-475e-8977-6d7ee57ee6d2 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:33:53.545977+0000 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:45:29.216 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.200+0000 7f1226ffd640 4 set_mon_vals no callback set 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.202+0000 7f122e98d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.202+0000 7f122e98d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.202+0000 7f122e98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.202+0000 7f122e98d640 1 -- 172.21.15.155:0/65047148 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1228075f10 con 0x7f122814a700 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.203+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 88820340 0 0) 0x7f1220003e40 con 0x7f122814a700 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.204+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 339942357 0 0) 0x7f1220084d90 con 0x7f122814a700 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.204+0000 7f12257fa640 10 client.4564.objecter ms_dispatch 0x7f1228000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.204+0000 7f12257fa640 3 client.4564.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.204+0000 7f12257fa640 3 client.4564.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.204+0000 7f12257fa640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.204+0000 7f1227fff640 10 monclient: get_auth_request con 0x7f120806db40 auth_method 0 2024-03-29T11:45:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.204+0000 7f12257fa640 10 client.4564.objecter ms_handle_connect 0x7f120806db40 2024-03-29T11:45:29.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.209+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1220050670 con 0x7f122814a700 2024-03-29T11:45:29.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.209+0000 7f12257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:29.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.209+0000 7f12257fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:29.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.531+0000 7f122e98d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:29.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.532+0000 7f122e98d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:29.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.532+0000 7f122e98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:45:29.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.532+0000 7f122e98d640 1 -- 172.21.15.155:0/65047148 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12280b4150 con 0x7f122814a700 2024-03-29T11:45:29.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.532+0000 7f12257fa640 1 -- 172.21.15.155:0/65047148 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1459448867) 0x7f12200525a0 con 0x7f122814a700 2024-03-29T11:45:29.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.532+0000 7f12257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:29.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.532+0000 7f12257fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:29.538 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:45:29.539 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"3bbabebe-1eb5-475e-8977-6d7ee57ee6d2","created":"2024-03-29T11:34:31.743246+0000","modified":"2024-03-29T11:44:36.686112+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:36.686112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e4ffdb02-aeee-44a2-8a4b-a09d7dc47ab6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"acce00ab-e768-4e25-a833-702e26eeda25","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"bca8f954-a16c-417f-a7c0-d4109884c11b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a27cee2b-5d5c-421f-bcc5-16b9e3f5b559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4422603-41d7-4505-9ab4-aafb96469ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"283819f5-206d-4020-9122-35eddb22cf6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"23b163db-4e0f-46de-b725-5d37debb31e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fe64015c-477d-4664-a96a-bc5ef201be2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:29.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.536+0000 7f122e98d640 10 librados: watch_flush enter 2024-03-29T11:45:29.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.536+0000 7f122e98d640 10 librados: watch_flush exit 2024-03-29T11:45:29.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.536+0000 7f122e98d640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-29T11:45:29.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.536+0000 7f122e98d640 10 client.4564.objecter successfully canceled tick 2024-03-29T11:45:29.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.536+0000 7f122e98d640 1 -- 172.21.15.155:0/65047148 >> v1:172.21.15.155:6816/1760814708 conn(0x7f120806db40 legacy=0x7f1208070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:29.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.536+0000 7f122e98d640 10 monclient: shutdown 2024-03-29T11:45:29.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.536+0000 7f122e98d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:29.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.536+0000 7f122e98d640 1 -- 172.21.15.155:0/65047148 >> v1:172.21.15.183:6789/0 conn(0x7f122814a700 legacy=0x7f12281e7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:29.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.537+0000 7f122e98d640 1 -- 172.21.15.155:0/65047148 shutdown_connections 2024-03-29T11:45:29.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.537+0000 7f122e98d640 1 -- 172.21.15.155:0/65047148 >> 172.21.15.155:0/65047148 conn(0x7f122806b660 msgr2=0x7f12280b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:29.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.537+0000 7f122e98d640 1 -- 172.21.15.155:0/65047148 shutdown_connections 2024-03-29T11:45:29.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.537+0000 7f122e98d640 1 -- 172.21.15.155:0/65047148 wait complete. 2024-03-29T11:45:29.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:45:29.537+0000 7f122e98d640 1 librados: shutdown 2024-03-29T11:45:29.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:29.558 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:45:29.560 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:45:29.560 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:45:29.561 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:29.596 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:45:29.596 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:45:29.596 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:29.674 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:45:29.675 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:45:29.675 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:29.752 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:45:29.753 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:45:29.753 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:29.844 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:45:29.845 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:45:29.845 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:29.894 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:45:29.894 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:45:29.894 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:29.932 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:45:29.932 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:45:29.932 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:30.014 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:45:30.014 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:45:30.015 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:30.083 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:45:30.083 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:45:30.083 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:45:30.083 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:30.108 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:45:30.108 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:45:30.108 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:30.147 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:45:30.147 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:45:30.147 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:45:30.147 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:30.159 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:45:30.159 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:45:30.159 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:30.211 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:45:30.212 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:45:30.212 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:30.223 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:45:30.223 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:45:30.223 DEBUG:teuthology.orchestra.run.smithi155:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:45:30.283 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:45:30.283 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:45:32.917 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:45:32.917 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:45:33.021 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:45:33.021 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:45:33.070 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:45:33.071 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:45:33.127 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:45:33.128 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:45:33.662 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:45:33.662 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:45:33.755 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:45:33.755 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:45:33.812 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:45:33.812 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:45:33.863 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:45:33.870 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710/data/mon.a.tgz 2024-03-29T11:45:33.871 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-29T11:45:33.892 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.EgbjZ1txUe 2024-03-29T11:45:33.892 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.EgbjZ1txUe 2024-03-29T11:45:34.004 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.EgbjZ1txUe 2024-03-29T11:45:34.098 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.EgbjZ1txUe is 126KB 2024-03-29T11:45:34.167 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.EgbjZ1txUe 2024-03-29T11:45:34.186 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710/data/mon.c.tgz 2024-03-29T11:45:34.186 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-29T11:45:34.249 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.39kxKj6Hgg 2024-03-29T11:45:34.250 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.39kxKj6Hgg 2024-03-29T11:45:34.341 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.39kxKj6Hgg 2024-03-29T11:45:34.440 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.39kxKj6Hgg is 126KB 2024-03-29T11:45:34.700 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.39kxKj6Hgg 2024-03-29T11:45:34.718 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710/data/mon.b.tgz 2024-03-29T11:45:34.718 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-29T11:45:34.742 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.DLqDYvcjlf 2024-03-29T11:45:34.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.DLqDYvcjlf 2024-03-29T11:45:34.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.DLqDYvcjlf 2024-03-29T11:45:34.941 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.DLqDYvcjlf is 126KB 2024-03-29T11:45:35.379 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.DLqDYvcjlf 2024-03-29T11:45:35.398 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:45:35.398 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:45:35.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:45:35.491 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:45:35.531 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:45:35.531 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:45:35.936 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710/remote/smithi155/crash 2024-03-29T11:45:36.113 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:45:36.155 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710/remote/smithi183/crash 2024-03-29T11:45:36.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:45:36.212 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:45:36.212 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:45:36.226 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:45:36.253 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40177.log 2024-03-29T11:45:36.253 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38325.log 2024-03-29T11:45:36.257 INFO:teuthology.orchestra.run.smithi155.stderr:gzip/var/log/ceph/ceph-client.admin.40177.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38676.log 2024-03-29T11:45:36.257 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39534.log 2024-03-29T11:45:36.257 INFO:teuthology.orchestra.run.smithi155.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40177.log.gz 2024-03-29T11:45:36.257 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38676.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38325.log.gzgzip 2024-03-29T11:45:36.257 INFO:teuthology.orchestra.run.smithi155.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:45:36.257 INFO:teuthology.orchestra.run.smithi155.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38676.log.gz 2024-03-29T11:45:36.257 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39534.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28524.log 2024-03-29T11:45:36.257 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40005.log 2024-03-29T11:45:36.257 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39534.log.gz 2024-03-29T11:45:36.257 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph.tmp-client.admin.28524.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28524.log.gz 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39768.log 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40295.log 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40005.log.gz 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40920.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39768.log.gz 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40959.log 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39339.log 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40959.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40295.log.gz 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39261.log 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39339.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40959.log.gz 2024-03-29T11:45:36.258 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40529.log 2024-03-29T11:45:36.300 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39888.log 2024-03-29T11:45:36.300 INFO:teuthology.orchestra.run.smithi155.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39339.log.gz 2024-03-29T11:45:36.300 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40529.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39261.log.gz 2024-03-29T11:45:36.300 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38637.log 2024-03-29T11:45:36.300 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40685.log 2024-03-29T11:45:36.300 INFO:teuthology.orchestra.run.smithi155.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40529.log.gz 2024-03-29T11:45:36.300 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40998.log 2024-03-29T11:45:36.300 INFO:teuthology.orchestra.run.smithi155.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39888.log.gz 2024-03-29T11:45:36.300 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40685.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38637.log.gz 2024-03-29T11:45:36.300 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41232.log 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40998.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40685.log.gz 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40334.log 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40646.log 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40998.log.gz 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40334.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41232.log.gz 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39690.log 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40646.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40334.log.gz 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40256.log 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38754.log 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40646.log.gz 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40412.log 85.3% 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39690.log.gz 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38754.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40256.log.gz 2024-03-29T11:45:36.301 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39417.log 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39495.log 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38754.log.gz 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39417.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40412.log.gzgzip 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41037.log 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39495.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39417.log.gz 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40083.log 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41466.log 86.3% 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39495.log.gz 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38793.log 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41037.log.gz 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41466.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40083.log.gz 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38598.log 2024-03-29T11:45:36.302 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38793.log: 86.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.41466.log.gz -5 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38167.log 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38598.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38793.log.gz 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38832.log 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38167.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38598.log.gz 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39612.log 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38832.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38167.log.gz 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40724.log 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39927.log 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38832.log.gz 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40724.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39612.log.gz 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39144.log 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39927.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40724.log.gz 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38089.log 2024-03-29T11:45:36.303 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41505.log 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39927.log.gz 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38206.log 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39144.log.gz 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39027.log 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38089.log.gz 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38206.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41505.log.gz 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39729.log 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39027.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 86.3% 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38206.log.gz 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39105.log 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39027.log.gz 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph.audit.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39729.log.gz 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38988.log 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39573.log 2024-03-29T11:45:36.304 INFO:teuthology.orchestra.run.smithi155.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38988.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39105.log.gz 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38520.log 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40451.log 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38988.log.gz 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38520.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39573.log.gz 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41154.log 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38520.log.gz 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.1.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40451.log.gz 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39066.log 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39651.log 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39066.log: gzip -5 --verbose 87.2% -- /var/log/ceph/ceph-client.admin.41271.log -- replaced with /var/log/ceph/ceph-client.admin.41154.log.gz 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:45:36.305 INFO:teuthology.orchestra.run.smithi155.stderr: 86.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.39066.log.gz 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38949.log 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39651.log: /var/log/ceph/ceph-client.admin.41271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38715.log 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39651.log.gz 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38949.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41271.log.gz 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39810.log 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40044.log 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38715.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38949.log.gz 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38128.log 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38715.log.gz 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38559.log 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39810.log.gz 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40373.log 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40044.log.gz 2024-03-29T11:45:36.306 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38128.log.gz 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.38559.log: /var/log/ceph/ceph-client.admin.40763.log 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41193.log 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38559.log.gz 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40763.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40373.log.gz 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39378.log 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39849.log 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40763.log.gz 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38403.log 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41193.log.gz 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39183.log 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39378.log.gz 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38403.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39849.log.gz 2024-03-29T11:45:36.307 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39183.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38403.log.gz 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38245.log 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40217.log 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.3.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39183.log.gz 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41427.log 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38245.log.gz 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38442.log 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38481.log 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40217.log.gz 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38442.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41427.log.gz 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38871.log 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38481.log: gzip -5 --verbose -- 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38442.log.gz 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr: /var/log/ceph/ceph-client.admin.41310.log 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38481.log.gz 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39300.log 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40881.log 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38871.log.gz 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39300.log: gzip -5 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41310.log.gz 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34737.log 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40881.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39300.log.gz 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.34737.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40881.log.gz 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34737.log.gz 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34634.log 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39222.log 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41076.log: /var/log/ceph/ceph-client.admin.34634.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39222.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41115.log 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34634.log.gz 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39222.log.gz 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40802.log 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38910.log/var/log/ceph/ceph-client.admin.41115.log: 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38284.log 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41115.log.gz 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40802.log.gz 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38910.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6%/var/log/ceph/ceph-client.admin.38284.log: -- replaced with /var/log/ceph/ceph-client.admin.38910.log.gz 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39456.log 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38284.log.gz 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40568.log 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39456.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:45:36.310 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39456.log.gz 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr: 92.7%/var/log/ceph/ceph-client.admin.40568.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38050.log 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr: 86.0%/var/log/ceph/ceph-osd.2.log: -- replaced with /var/log/ceph/ceph-client.admin.40568.log.gz 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41349.log 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38364.log 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38050.log.gz 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41349.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41349.log.gz 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40137.log 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40490.log 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40137.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38364.log.gz 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40607.log 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5%/var/log/ceph/ceph-client.admin.40490.log: -- replaced with /var/log/ceph/ceph-client.admin.40137.log.gz 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40490.log.gz 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr: -- /var/log/ceph/ceph-client.admin.41388.log 2024-03-29T11:45:36.311 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39966.log 2024-03-29T11:45:36.312 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41388.log: /var/log/ceph/ceph-client.admin.40607.log: 87.3% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40607.log.gz 2024-03-29T11:45:36.312 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41388.log.gz 2024-03-29T11:45:36.312 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40841.log 2024-03-29T11:45:36.312 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34343.log 2024-03-29T11:45:36.312 INFO:teuthology.orchestra.run.smithi155.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39966.log.gz/var/log/ceph/ceph-client.admin.40841.log: 2024-03-29T11:45:36.312 INFO:teuthology.orchestra.run.smithi155.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40841.log.gz 2024-03-29T11:45:36.312 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:45:36.312 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.34343.log: /var/log/ceph/ceph-mon.c.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34343.log.gz 2024-03-29T11:45:36.314 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:45:36.314 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34510.log 2024-03-29T11:45:36.314 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34470.log 2024-03-29T11:45:36.314 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34630.log 2024-03-29T11:45:36.314 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34470.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34510.log.gz 2024-03-29T11:45:36.314 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:45:36.314 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34630.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34470.log.gz 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.audit.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34630.log.gz 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34710.log 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.4.log: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gzgzip 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34670.log 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.7.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34710.log.gz 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34590.log 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34670.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34430.log 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34670.log.gz 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34590.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34430.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34590.log.gz 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:45:36.315 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mgr.x.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34430.log.gz 2024-03-29T11:45:36.316 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34550.log 2024-03-29T11:45:36.316 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:45:36.316 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34550.log: 87.2%/var/log/ceph/ceph-mon.b.log: -- replaced with /var/log/ceph/ceph.log.gz 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34550.log.gz 2024-03-29T11:45:36.316 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:45:36.316 INFO:teuthology.orchestra.run.smithi183.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi155.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:45:48.113 INFO:teuthology.orchestra.run.smithi155.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:45:48.211 INFO:teuthology.orchestra.run.smithi183.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:45:48.299 INFO:teuthology.orchestra.run.smithi155.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:45:48.675 INFO:teuthology.orchestra.run.smithi183.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:45:50.983 INFO:teuthology.orchestra.run.smithi183.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:45:52.319 INFO:teuthology.orchestra.run.smithi155.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:45:52.443 INFO:teuthology.orchestra.run.smithi155.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:45:52.615 INFO:teuthology.orchestra.run.smithi183.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:46:27.079 INFO:teuthology.orchestra.run.smithi155.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:46:32.081 INFO:teuthology.orchestra.run.smithi155.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:46:32.082 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:46:32.083 INFO:teuthology.orchestra.run.smithi155.stderr:real 0m55.843s 2024-03-29T11:46:32.083 INFO:teuthology.orchestra.run.smithi155.stderr:user 2m36.927s 2024-03-29T11:46:32.083 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m5.576s 2024-03-29T11:46:42.861 INFO:teuthology.orchestra.run.smithi183.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:46:42.862 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:46:42.862 INFO:teuthology.orchestra.run.smithi183.stderr:real 1m6.588s 2024-03-29T11:46:42.863 INFO:teuthology.orchestra.run.smithi183.stderr:user 1m57.300s 2024-03-29T11:46:42.863 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m4.390s 2024-03-29T11:46:42.863 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:46:42.863 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710/remote/smithi155/log 2024-03-29T11:46:42.864 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:47:06.778 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710/remote/smithi183/log 2024-03-29T11:47:06.779 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:47:25.617 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:47:25.828 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cc5a7c76b96c4ee8b69a22b4f9cd5441 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:47:25.832 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:47:25.848 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:47:25.881 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:47:25.882 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:47:25.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:47:26.006 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:47:26.025 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:47:26.026 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:47:26.029 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:47:26.044 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:47:26.047 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:47:26.050 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:47:26.051 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-29T11:47:26.051 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +420us[ +420us] +/- 72ms 2024-03-29T11:47:26.052 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -121us[ -116us] +/- 68ms 2024-03-29T11:47:26.052 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 -229us[ -229us] +/- 67ms 2024-03-29T11:47:26.053 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:47:26.055 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:47:26.056 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-29T11:47:26.056 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 +394us[ +394us] +/- 72ms 2024-03-29T11:47:26.057 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -134us[ -140us] +/- 68ms 2024-03-29T11:47:26.057 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 -237us[ -237us] +/- 67ms 2024-03-29T11:47:26.057 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:47:26.058 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:47:26.080 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:47:26.081 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:47:26.102 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:47:26.175 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:47:26.249 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:47:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:26.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:47:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:26.338 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:47:26.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:47:26.594 INFO:teuthology.task.internal:Duration was 1266.790138 seconds 2024-03-29T11:47:26.594 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:47:27.090 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:47:27.092 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:47:27.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:47:27.142 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:47:27.145 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:47:27.577 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:47:27.577 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:47:27.577 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:47:27.601 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:47:27.601 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:47:27.631 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:47:27.631 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:47:27.645 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:47:27.706 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:47:27.706 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:47:27.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:47:27.850 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:47:27.892 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:47:27.892 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:47:27.921 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:47:27.949 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:47:27.960 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:27.964 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:27.987 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-03-29T11:47:28.015 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-03-29T11:47:28.040 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:28.090 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:28.105 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:47:28.115 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:47:28.116 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710/remote/smithi155 2024-03-29T11:47:28.116 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:47:28.167 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710/remote/smithi183 2024-03-29T11:47:28.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:47:28.203 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:47:28.204 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:47:28.207 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:47:28.253 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:47:28.266 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:47:28.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:47:28.341 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:47:28.341 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:47:28.344 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:47:28.356 INFO:teuthology.orchestra.run.smithi155.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:47 /home/ubuntu/cephtest 2024-03-29T11:47:28.358 INFO:teuthology.orchestra.run.smithi183.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:47 /home/ubuntu/cephtest 2024-03-29T11:47:28.360 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:47:28.370 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:47:28.437 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:47:28.465 DEBUG:teuthology.nuke:shortname: smithi155 2024-03-29T11:47:28.466 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:47:28.493 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:37.026359', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHV5Cxb2pgYCJ71n+4eseDt81EMdVU/jdelCc8I0IKUkVzbtbHdsAdlkS3a1HSHERpFUUwZ5c9F6VwCnfo9B8uU='} 2024-03-29T11:47:28.497 DEBUG:teuthology.nuke:shortname: smithi183 2024-03-29T11:47:28.497 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:47:28.521 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622710', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:37.027963', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGlI3AOi5gl6ENZptPZ3nn1/HIM53rjn8vZ9KNQE9ERnA1k0FrtNBNenc01NO2wr+dWLmdyahceIHz3EWoFGyI='} 2024-03-29T11:47:28.559 INFO:teuthology.orchestra.console.smithi155:Power off 2024-03-29T11:47:28.559 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:47:28.585 DEBUG:teuthology.orchestra.console.smithi155:power off output: Chassis Power Control: Down/Off 2024-03-29T11:47:28.586 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:28.604 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-29T11:47:28.604 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:47:28.619 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-29T11:47:28.631 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-29T11:47:28.632 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:28.658 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-29T11:47:32.620 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:32.660 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:32.738 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-29T11:47:32.773 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-29T11:47:36.742 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:36.774 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:36.854 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is off 2024-03-29T11:47:36.886 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-29T11:47:36.956 INFO:teuthology.orchestra.console.smithi155:Power off completed 2024-03-29T11:47:36.987 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-29T11:47:37.092 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1266.7901384830475 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cc5a7c76b96c4ee8b69a22b4f9cd5441 status: fail success: false 2024-03-29T11:47:37.092 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:47:37.180 INFO:teuthology.run:FAIL